acceptodds
Under review as a conference paper at ICLR 2027

Beyond Scores with Reward-Model Embeddings for Diverse Flow RL

Abstract

Reinforcement learning (RL) improves text-to-image flow models, but scalar rewards assigned independently to individual images overlook visual redundancy within a prompt group. We show that reward-model embeddings provide diversity feedback within each prompt group. We propose using regularized D-optimal design to compute diversity weights from reward-model embeddings and reshape scalar rewards before advantage computation. Our method requires no additional image encoder or encoder forward pass and preserves the underlying sampling and policy-update rules. We evaluate our method on Stable Diffusion 3.5 Medium (SD3.5-M) with Advantage Weighted Matching (AWM) and DiffusionNFT, two RL algorithms for flow models. On DrawBench, our method improves overall quality and within-prompt diversity, including gains on three held-out reward models not used during training. On Pick-a-Pic, our method attains each baseline's highest evaluation reward faster for AWM and faster for DiffusionNFT, measured in GPU hours.

Primary area:
generative models
Keywords:
reinforcement learning, flow models, reward shaping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d4j5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at within-prompt diversity collapse in reinforcement-learning fine-tuning of text-to-image flow models. The proposed method reuses image embeddings already computed by a reward model and solves a regularized D-optimal design problem within each prompt group. Samples that contribute complementary embedding directions get larger weights, whereas redundant samples share less weight. These weights multiplicatively reshape scalar rewards before advantage computation, without changing the sampler or the policy-update rule. The method is evaluated with AWM and DiffusionNFT on SD3.5 Medium.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The method is lightweight and has a clean interface: it reuses embeddings already produced during reward evaluation.
Critical weaknesses
  1. The diveristy score is achieved by utilizing embedding from the reward model. It may constrain the deployment for the proposed method. For example, in some rule-based reward setting, there is no entrance to obtain these embeddings unless bringing in another differentiable reward model and conducting another evaluation.

  2. I wonder whether diveristy score will be influenced by other factors, e.g., noises or artifacts in the images. Considering that the integration of diveristy score is only by reweighting, it is hard to judge whether the goal of deploying reweighting really aligns with the goal of enhancing image rewards.

Questions for authors
  1. Please see weaknesses.
  2. Please provide some diversity-related metrics to report the exact change after deploying the proposed method.
  3. In addition, some related works that discuss process rewards or conduct similarly by reweighting techniques could also be inserted and discussed in the related works, e.g, [1,2,3].
  4. At this time, I lean to accept for this paper. However, some concerns remain unsolved and I will defer my final decision after rebuttal.

[1] He X, Fu S, Zhao Y, et al. Tempflow-grpo: When timing matters for grpo in flow models[C]//International Conference on Learning Representations. 2026, 2026: 132869-132895.

[2] Tong Y, Liu M, Zhao C, et al. Alleviating sparse rewards by modeling step-wise and long-term sampling effects in flow-based grpo[J]. arXiv preprint arXiv:2602.06422, 2026.

[3] Deng H, Yan K, Mao C, et al. Densegrpo: From sparse to dense reward for flow matching model alignment[J]. arXiv preprint arXiv:2601.20218, 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer d4j5 (1/2): W1 and Q1

Thank you for your positive assessment and helpful feedback. We address each point below.

W1 & Q1

W1. The diveristy score is achieved by utilizing embedding from the reward model. It may constrain the deployment for the proposed method. For example, in some rule-based reward setting, there is no entrance to obtain these embeddings unless bringing in another differentiable reward model and conducting another evaluation.

Q1. Please see weaknesses.

Response: Thank you for raising this practical point. Rule-based rewards do not require a differentiable reward model under our method, because we compute diversity weights before advantage calculation and never backpropagate gradients through the encoder. Since pure rule-based rewards do not provide embeddings, applying our method requires one frozen forward pass from an off-the-shelf vision encoder (such as CLIP or DINOv3). With learned reward models, reusing their internal embeddings adds zero extra passes. In our separate encoder ablation on DrawBench, where diversity weights are applied directly to advantages under the same joint reward objective, different frozen encoders consistently improve overall quality over that ablation's baseline:

Diversity EncoderAWM Overall Quality Gain (%)DiffusionNFT Overall Quality Gain (%)
None (Baseline)0.000.00
PickScore+6.37+2.98
CLIPScore+6.42+2.90
HPSv2.1+5.68+2.19

Because our experiments evaluate learned reward models only, we have noted the extra frozen encoder pass needed for rule-based rewards in the Limitations section.

AuthorsOpenReview ↗

Response to Reviewer d4j5 (2/2): W2 and Q2–Q3

W2 & Q2

W2. I wonder whether diveristy score will be influenced by other factors, e.g., noises or artifacts in the images.

Response: Noisy or artifact-corrupted images can produce unusual embeddings, but multiplicative shaping prevents our method from favoring them. An additive bonus rewards visually different images even when their quality is low. Multiplicative shaping instead multiplies the diversity weight by the reward score, so it boosts diverse images only when their quality is high.

W2. Considering that the integration of diveristy score is only by reweighting, it is hard to judge whether the goal of deploying reweighting really aligns with the goal of enhancing image rewards.

Response: Diversity reweighting aligns with improving image rewards because it prevents early collapse onto repeated copies of a single high-scoring image during RL training. Within each prompt group, regularized D-optimal weights divide one sample's weight across near-duplicates so copies do not dominate advantage estimation, while maximizing the log-determinant expands the effective support of the conditional distribution of images given the prompt across distinct high-reward modes. To the best of our knowledge, there is no theoretical analysis in RL showing that promoting diversity improves optimization of the original reward, although many empirical works such as GARDO report such gains. Along these lines, we bring tools from diversity to flow RL and show consistent empirical gains on the original reward. At test time, all methods are compared using existing reward models with unmodified rewards. Our main reward-reshaping method improves mean DrawBench quality across all six reward models and speeds up training. On Pick-a-Pic, we set the target to each baseline's highest evaluation reward within 60 GPU hours. Training speedup is the ratio between the GPU hours the baseline needs and the GPU hours our method needs to reach this target:

MethodDrawBench Quality Gain (%)Training Speedup
AWM + Ours+2.621.38×
DiffusionNFT + Ours+3.384.32×

We have noted this empirical scope in the Limitations section.

Q2. Please provide some diversity-related metrics to report the exact change after deploying the proposed method.

Response: Thank you for this suggestion. Our method increases conditional visual diversity over the baseline under both AWM and DiffusionNFT. Following the main evaluation setup in our paper with SD3.5-M and AWM on DrawBench, we compute each metric over the 4 images conditioned on each prompt and average the scores across all 200 prompts. PickScore diversity measures cosine distance to the nearest neighbor in the prompt group, DreamSim measures pairwise perceptual distance learned from human similarity judgments, and the Vendi score computes the effective number of distinct visual modes from DreamSim embeddings:

MethodPickScore Diversity (×10⁻³)DreamSimVendi
AWM6.300.23101.467
AWM + GARDO5.910.26801.535
AWM + Ours9.180.29551.591

Under DiffusionNFT, our method also increases PickScore diversity by 8.45% over the baseline, from 9.61 × 10⁻³ to 10.42 × 10⁻³. We have included these held-out diversity metrics in the paper.

Q3

Q3. In addition, some related works that discuss process rewards or conduct similarly by reweighting techniques could also be inserted and discussed in the related works, e.g, [1,2,3].

[1] He X, Fu S, Zhao Y, et al. Tempflow-grpo: When timing matters for grpo in flow models[C]//International Conference on Learning Representations. 2026, 2026: 132869-132895.

[2] Tong Y, Liu M, Zhao C, et al. Alleviating sparse rewards by modeling step-wise and long-term sampling effects in flow-based grpo[J]. arXiv preprint arXiv:2602.06422, 2026.

[3] Deng H, Yan K, Mao C, et al. Densegrpo: From sparse to dense reward for flow matching model alignment[J]. arXiv preprint arXiv:2601.20218, 2026.

Response: Thank you for pointing us to these related works. TempFlow-GRPO [1], Tong et al. [2], and DenseGRPO [3] improve temporal credit assignment along the denoising trajectory through timing-aware branching, step-wise sampling effects, and dense step-level rewards. Our method instead reweights terminal rewards across images within a prompt group to reduce visual redundancy, so it is complementary to trajectory-level credit assignment. We have cited and discussed all three works in the Related Work section.

Reviewer_d4j5OpenReview ↗

Response to Authors

Thank you for your responses.

W1: My concern is that computing diversity-induced weights requires an embedding-model forward pass and depends on its representation quality. In rule-based settings such as the OCR task in Flow-GRPO, an additional vision encoder is needed, yet its representations may not capture task-relevant semantics.

W2: Could you clarify where the bonus reward mentioned in your response is described in the paper? While diversity matters, I remain uncertain whether reweighting without additional supervision improves performance or task alignment. As this concern partly reflects my own experience, I accept author's explanation for now. No further response on this specific issue is needed. :)

I have not found substantive revisions addressing my concerns, particularly in the limitations and related work sections. Concrete manuscript revisions—not rebuttal commitments alone—are needed. Unresolved concerns may negatively affect my final assessment. I will maintain my current rating provisionally and defer my final rating until I have considered the other reviewers’ feedback and the subsequent discussion.

Reviewer_gKUbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the diversity problem of flow matching during RL fine-tuning: in text-to-image RL pipelines, reward models typically assign scalar score to each image independently, which may cause the model to repeatedly reinforce certain similar samples and reduce within-prompt diversity. The core finding is that the reward model is not only a quality scorer, its image embeddings can also serve as a diversity scorer. To this end, the authors propose a reward reshaping method that uses regularized D-optimal design to solve for diversity weights on the probability simplex and uses the post-processed weights to reshape rewards before computing advantages. To reduce computation, the authors further use SVD to reduce the D-optimal problem to a lower-dimensional space for solving. Experimental results show that the method improves both generation quality and within-prompt diversity.

Critical strengths
  1. Reusing the reward model's embeddings as diversity signals avoids training an additional diversity encoder. The method changes only the rewards and does not alter sampling or policy-update rules, so it can in principle support various flow RL algorithms.
  2. The theoretical tools are reasonable: regularized D-optimal design maximizes coverage in the embedding space and has a clear geometric interpretation. The SVD reduction and Frank-Wolfe solver also take computational feasibility into account.
Critical weaknesses
  1. The assumptions about the task setting lack empirical support, and the solution is rather heuristic and lacks theoretical grounding. The authors claim: "The reward model can therefore continue to assign high scores to visually similar images." However, they do not explicitly analyze how high the repetition rate is among high-scoring images in practice. Moreover, they do not provide an error estimate of how much adjusting weights by maximizing embedding coverage affects the original RL training process.
  2. The experimental section has shortcomings: 1) Generalization validation is limited. The method is tested only on SD3.5-M and two forward-matching RL algorithms, without experiments on reverse-trajectory RL. 2) The evaluation dataset and statistical result are insufficient. DrawBench contains only 200 evaluation prompts, the number of evaluated samples is limited. No multiple random seeds, confidence intervals, or significance tests are reported. 3) Qualitative experiments are insufficient. The authors show only a few comparison examples, and the prompts used are relatively simple. It is suggested to add a user study and enrich the evaluated prompts.
Questions for authors
  1. By strategically adjusting advantage scores to increase output diversity, the authors need to analyze the impact of D-optimal weights on policy gradients, advantage estimation, and the optimality of the original reward, and explain when diversity can be improved without harming quality and when bias may be introduced.
  2. It is suggested to add validation results on reverse-trajectory methods such as Flow-GRPO, only then can the conclusion that the method "can be applied to different RL methods" be truly supported.
  3. Supplement the qualitative experiments by showing failure cases on complex prompts, counting, text rendering, spatial relations, and other tasks, and discuss whether improved diversity may bring prompt inconsistency, bias amplification, or safety risks.
6 comments
AuthorsOpenReview ↗

Response to Reviewer gKUb (1/3): W1 and generalization

Thank you for your thorough review and suggestions. We address each weakness and question below.

W1

W1. The assumptions about the task setting lack empirical support ... The authors claim: "The reward model can therefore continue to assign high scores to visually similar images." However, they do not explicitly analyze how high the repetition rate is among high-scoring images in practice.

Response: Thank you for pointing this out. We agree that our paper needed a direct empirical check on visual repetition among high-scoring images. Across all 200 DrawBench prompts, we generate 4 images per prompt with pretrained SD3.5-M and score each image with our combined PickScore, CLIPScore, and HPSv2.1 reward before and after full D-optimal reweighting. Using DreamSim to measure pairwise perceptual similarity within each group, we report how often the two highest-reward images are also the most similar:

Reward WeightingHighest-Reward Images Are Most Similar (%)
Before reweighting35.5
After full D-optimal reweighting22.5

Since each 4-image group has 6 image pairs, this rate would be 16.7% by random chance compared with our measured 35.5%, which means high-scoring pretrained generations frequently repeat similar visuals within a prompt group. Full D-optimal reweighting substantially reduces this overlap from 35.5% to 22.5%.

We have revised our Introduction statement to clarify that visually similar images can receive similarly high rewards when each satisfies the prompt, and we have added this analysis in the appendix.

W1. ... and the solution is rather heuristic and lacks theoretical grounding. ... Moreover, they do not provide an error estimate of how much adjusting weights by maximizing embedding coverage affects the original RL training process.

Response: Our weighting rule is grounded in regularized D-optimal design, a well-understood statistical criterion that maximizes log-determinant coverage over group embeddings, while we do not claim a theoretical error bound on RL training. To the best of our knowledge, there is no theoretical analysis in RL showing that promoting diversity improves optimization of the original reward, although many empirical works such as GARDO report such gains. Along these lines, we bring tools from diversity to flow RL and show consistent empirical gains on the original reward. We have noted this empirical scope in the Limitations section.

W2 & Q2

W2. The experimental section has shortcomings: 1) Generalization validation is limited. The method is tested only on SD3.5-M ...

Response: Our method generalizes beyond SD3.5-M to FLUX.2-klein-base-4B and improves all three training rewards and aggregate reward over the DiffusionNFT baseline on DrawBench. Following the same DiffusionNFT training and DrawBench evaluation setup as our main SD3.5-M experiment in the paper, we change only the backbone from SD3.5-M to FLUX.2-klein-base-4B. Aggregate reward is the scaled sum of the three rewards, and reward gain is their mean relative improvement:

MethodPickScoreCLIPScoreHPSv2.1Aggregate RewardReward Gain (%)
FLUX.2-klein-base-4B (Pretrained)20.510.2390.2041.2316-
DiffusionNFT23.060.2660.3171.47040.00
DiffusionNFT + Ours23.510.2770.3341.5150+3.72

We have included these results in the paper.

W2. ... and two forward-matching RL algorithms, without experiments on reverse-trajectory RL.

Q2. It is suggested to add validation results on reverse-trajectory methods such as Flow-GRPO, only then can the conclusion that the method "can be applied to different RL methods" be truly supported.

Response: Thank you for raising this point. We agree that saying our method "can be applied to different RL methods" was too broad. Our current method applies only to two forward-matching RL algorithms, AWM and DiffusionNFT. Both methods represent the state of the art in flow models, and their papers report order-of-magnitude improvements in statistical efficiency over Flow-GRPO. We have narrowed our claims in the paper and the Limitations section to state that our current approach is limited to forward-matching flow RL.

AuthorsOpenReview ↗

Response to Reviewer gKUb (2/3): W2 continued

W2 & Q2 (continued)

W2. 2) The evaluation dataset and statistical result are insufficient. DrawBench contains only 200 evaluation prompts, the number of evaluated samples is limited. No multiple random seeds, confidence intervals, or significance tests are reported.

Response: Thank you for this suggestion. Across all 2,048 prompts in the held-out Pick-a-Pic test split, our method improves every reward metric and aggregate reward over both the baseline without reweighting and GARDO under both AWM and DiffusionNFT. Following the same SD3.5-M training setup as in our paper, we change only the evaluation set from the 200 DrawBench prompts used in GARDO, DiffusionNFT, and AWM to the 2,048 held-out Pick-a-Pic test prompts. Aggregate reward is the scaled sum of the three rewards, and reward gain is their mean relative improvement:

MethodPickScoreCLIPScoreHPSv2.1Aggregate RewardReward Gain (%)
AWM22.750.29270.33781.50560.00
AWM + GARDO21.690.28190.32411.4403-4.14
AWM + Ours23.180.30240.35341.5475+3.28
DiffusionNFT22.840.28620.34751.51220.00
DiffusionNFT + GARDO22.960.28660.35381.5235+0.82
DiffusionNFT + Ours23.310.29460.36321.5545+3.17

Because each full RL run takes roughly 60 GPU hours, we follow prior works on RL for flow models such as GARDO, DiffusionNFT, and AWM in reporting single-seed training runs. We mitigate this by evaluating across two RL algorithms (AWM and DiffusionNFT), two text-to-image models (SD3.5-M and FLUX.2-klein-base-4B, where our method achieves a 3.72% DrawBench reward gain), and six training and held-out reward models. Our method achieves consistent gains across all of them. We have included these results in the paper.

W2. 3) Qualitative experiments are insufficient. The authors show only a few comparison examples, and the prompts used are relatively simple. It is suggested to ... enrich the evaluated prompts.

Response: Thank you for this suggestion. We have added qualitative comparisons on more demanding text-rendering and multi-element prompts in the appendix. We find that our method preserves prompt fidelity while increasing visual variation over the AWM baseline. For example, on a storefront sign prompt, our method renders "Hello World" accurately in all four images with varied text placement, compared with one baseline image. On an underwater-elephant prompt, both methods match the prompt in all four images while ours varies pose, lighting, and surroundings.

W2. ... It is suggested to add a user study ...

Response: We leave a controlled user study to future work because conducting a well-designed human study is not feasible within the rebuttal window. Following standard practice in prior flow-RL works [1, 2, 3], we evaluate human preference and prompt faithfulness on DrawBench with three held-out reward models excluded from training (ImageReward, UnifiedReward, and Aesthetic), and our method improves all three under both AWM and DiffusionNFT:

MethodImageRewardUnifiedRewardAesthetic
AWM1.423.346.01
AWM + Ours1.503.406.15
DiffusionNFT1.423.366.19
DiffusionNFT + Ours1.523.476.22

[1] He, Haoran, et al. "GARDO: Reinforcing Diffusion Models without Reward Hacking." NeurIPS 2026.

[2] Zheng, Kaiwen, et al. "DiffusionNFT: Online Diffusion Reinforcement with Forward Process." ICLR 2026.

[3] Xue, Shuchen, et al. "Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models." ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer gKUb (3/3): Q1 and Q3

Q1

Q1. By strategically adjusting advantage scores to increase output diversity, the authors need to analyze the impact of D-optimal weights on policy gradients, advantage estimation, and the optimality of the original reward, and explain when diversity can be improved without harming quality and when bias may be introduced.

Response: Regularized D-optimal design removes duplicates through reweighting: it optimizes all group weights jointly and splits one sample's weight across duplicate images in the prompt group, so repeated generations do not dominate advantage estimation or policy gradients. From a distributional view, maximizing the log-determinant increases the effective support of the conditional distribution of images given the prompt. Bias can arise when a prompt has only one narrow valid answer and strong reweighting boosts visually different lower-quality images, which our multiplicative shaping keeps small. To the best of our knowledge, there is no theoretical analysis in RL showing that promoting diversity improves optimization of the original reward, although many empirical works such as GARDO report such gains. Along these lines, we bring tools from diversity to flow RL and show consistent empirical gains on the original reward. We have noted this empirical scope in the Limitations section.

Q3

Q3. Supplement the qualitative experiments by showing failure cases on complex prompts, counting, text rendering, spatial relations, and other tasks ...

Response: Thank you for this suggestion. We have added a failure-case analysis on complex prompts in the appendix. We find that diversity weighting does not explicitly enforce exact object counts or spatial relations, so our method can still fail on those constraints even while improving text rendering such as "Hello World" across all four storefront images. For example, our method misses exact object counts on a prompt asking for three cats and two dogs, and sometimes places a cat on the wrong side of a racket.

Q3. ... and discuss whether improved diversity may bring prompt inconsistency, bias amplification, or safety risks.

Response: We have added a short discussion in the Limitations section. Diversity weighting does not explicitly enforce prompt constraints, may amplify biases in pretrained reward models, and does not filter unsafe content. We have not evaluated its effects on bias or safety.

Reviewer_WJqaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes reward reshaping for diverse flow-model RL. It extracts per-image embeddings from an existing reward model, solves a regularized D-optimal-design allocation within each prompt group, and multiplies the scalar rewards by the resulting mean-one diversity weights before recomputing advantages. The module is applied to AWM and DiffusionNFT on SD3.5 Medium. The authors report improvements in six quality metrics, a PickScore-embedding diversity metric, and baseline-relative training speed on DrawBench/Pick-a-Pic.

Critical strengths

S1: The paper addresses a real blind spot of scalar per-sample reward optimization: it does not directly distinguish a set of high-quality but redundant generations from a set with wider coverage.

S2: The method is concise and implementation-friendly. Reusing reward-model features avoids another encoder pass, and the SVD reduction makes the D-optimal calculation tractable for the small prompt groups used in RL rollouts.

Critical weaknesses

W1: The diversity evidence is not sufficiently independent. The central metric is based on normalized PickScore embeddings, and the proposed method explicitly reweights samples using reward-model embeddings (including PickScore in the main configuration). This can mechanically optimize the measurement geometry without showing that humans perceive a more diverse, useful, or prompt-faithful set. Independent feature spaces, pairwise human diversity judgments, and coverage/mode-collapse metrics are needed.

W2: The method's causal contribution is under-isolated. The paper compares D-optimal reweighting to baseline and GARDO, but does not include simple controls such as uniform noise/entropy regularization, farthest-point or cosine-distance reweighting, a pairwise repulsion loss, or a rank-preserving additive bonus. Consequently, it is unclear whether D-optimal design itself is necessary rather than any group-level anti-redundancy heuristic.

W3: Multiplying raw scalar rewards by weights is a consequential and under-justified design decision. It can change within-group rankings and alter reward scale in ways entangled with group-normalized advantages, especially across reward models with different offsets/scales. The paper does not ablate multiplicative versus additive/rank-preserving shaping, the ridge/interpolation strength, or whether the quality improvements remain after explicitly constraining the original mean reward.

W4: The claim that reward-model embeddings are a general diversity signal is stronger than the evidence. Such embeddings may encode aesthetic/style variation, semantic content, or reward-model biases, not necessarily the diversity users want; rewarding a direction that is merely unusual can conflict with prompt fidelity. The paper needs targeted failure analysis and task-dependent diversity definitions.

Questions for authors

Q1: Can the authors evaluate diversity with feature extractors excluded from both training and method selection, plus human pairwise set-diversity and prompt-faithfulness judgments?

Q2: How does the proposed allocation compare with simple cosine-distance/farthest-point weighting, entropy bonuses, and rank-preserving/additive reward shaping under matched rollout and tuning budgets?

Q3: Please ablate the ridge coefficient, reshaping interpolation strength, group size, and multiplicative versus additive shaping. Does the original scalar reward remain unchanged or improve when it is explicitly constrained?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WJqa (1/3): W1 and Q1

Thank you for your constructive review and detailed questions. We address each point below.

W1 & Q1

W1. The diversity evidence is not sufficiently independent. The central metric is based on normalized PickScore embeddings, and the proposed method explicitly reweights samples using reward-model embeddings (including PickScore in the main configuration). This can mechanically optimize the measurement geometry ... Independent feature spaces ... and coverage/mode-collapse metrics are needed.

Q1. Can the authors evaluate diversity with feature extractors excluded from both training and method selection ...?

Response: Thank you for this suggestion. On two held-out diversity metrics excluded from both training and method selection, our method achieves higher conditional perceptual diversity and mode coverage than both the AWM baseline and AWM + GARDO outside the PickScore feature space. Following the main evaluation setup in our paper with SD3.5-M and AWM on DrawBench, we compute each metric over the 4 images conditioned on each prompt and average the scores across all 200 prompts. DreamSim measures pairwise perceptual distance learned from human similarity judgments, and the Vendi score computes the effective number of distinct visual modes from DreamSim embeddings:

MethodDreamSimVendi
AWM0.23101.467
AWM + GARDO0.26801.535
AWM + Ours0.29551.591

We have included these held-out diversity metrics in the paper.

W1. ... This can mechanically optimize the measurement geometry without showing that humans perceive a more diverse, useful, or prompt-faithful set. ... pairwise human diversity judgments ... are needed.

Q1. Can the authors evaluate ... human pairwise set-diversity and prompt-faithfulness judgments?

Response: Our method improves human preference and prompt faithfulness on DrawBench across three held-out reward models excluded from training (ImageReward, UnifiedReward, and Aesthetic) under both AWM and DiffusionNFT, as in the evaluation protocol of prior flow-RL works [1, 2, 3]:

MethodImageRewardUnifiedRewardAesthetic
AWM1.423.346.01
AWM + Ours1.503.406.15
DiffusionNFT1.423.366.19
DiffusionNFT + Ours1.523.476.22

We agree that a controlled human study is a worthwhile direction, and we leave it to future work because running a well-designed user study is not feasible within the rebuttal window.

[1] He, Haoran, et al. "GARDO: Reinforcing Diffusion Models without Reward Hacking." NeurIPS 2026.

[2] Zheng, Kaiwen, et al. "DiffusionNFT: Online Diffusion Reinforcement with Forward Process." ICLR 2026.

[3] Xue, Shuchen, et al. "Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models." ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer WJqa (2/3): W2 and Q2

W2 & Q2

W2. The method's causal contribution is under-isolated. The paper compares D-optimal reweighting to baseline and GARDO, but does not include simple controls such as uniform noise/entropy regularization, farthest-point or cosine-distance reweighting, a pairwise repulsion loss, or a rank-preserving additive bonus. Consequently, it is unclear whether D-optimal design itself is necessary rather than any group-level anti-redundancy heuristic.

Q2. How does the proposed allocation compare with simple cosine-distance/farthest-point weighting, entropy bonuses, and rank-preserving/additive reward shaping under matched rollout and tuning budgets?

Response: We do not claim that D-optimal design is the only global criterion, and other group-level anti-redundancy criteria could also work. We use regularized D-optimal design because it is a simple algorithm that reduces redundant images globally across a prompt group with negligible overhead. By contrast, the closest prior work on RL for diversity (GARDO) uses nearest-neighbor cosine distance, which scores each image using only its distance to its single closest neighbor and ignores the rest of the group. D-optimal design instead optimizes all group weights jointly and splits the weight of one sample across duplicate images in the prompt group.

Following the training and evaluation setup in our paper, we compare three reweighting methods on DrawBench: uniform weighting (the baseline without reweighting), nearest-neighbor cosine-distance reweighting (GARDO), and D-optimal reweighting (Ours). For nearest-neighbor cosine-distance reweighting, we replace each sample's D-optimal diversity score with its cosine distance to its nearest neighbor in the prompt group. We measure overall quality gain as the mean relative improvement across all six reward models. Our method achieves the highest overall quality gain on both AWM and DiffusionNFT:

MethodReweighting MethodPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedRewardOverall Quality Gain (%)
AWMUniform (Baseline)23.410.2980.3176.011.423.340.00
AWMCosine (GARDO)22.690.2860.3165.831.373.18-3.09
AWMD-optimal (Ours)23.690.2970.3326.151.503.40+2.62
DiffusionNFTUniform (Baseline)23.340.2840.3276.191.423.360.00
DiffusionNFTCosine (GARDO)23.460.2850.3336.121.423.44+0.61
DiffusionNFTD-optimal (Ours)23.700.2920.3446.221.523.47+3.38

Entropy bonuses, farthest-point weighting, and pairwise repulsion losses are also reasonable alternatives, but each full RL run takes roughly 60 GPU hours. We therefore focused on uniform and cosine-distance baselines here and leave broader heuristic comparisons to future work. We have clarified the distinction between D-optimal and nearest-neighbor reweighting in the paper.

AuthorsOpenReview ↗

Response to Reviewer WJqa (3/3): W3–W4 and Q3

W3 & Q3

W3. Multiplying raw scalar rewards by weights is a consequential and under-justified design decision. It can change within-group rankings and alter reward scale in ways entangled with group-normalized advantages, especially across reward models with different offsets/scales. The paper does not ablate multiplicative versus additive/rank-preserving shaping ...

Q3. Please ablate ... multiplicative versus additive shaping.

Response: Multiplicative shaping outperforms an additive diversity-bonus variant in evaluation reward under both AWM and DiffusionNFT. We use the same training and evaluation settings as in the paper, and report the average combined reward of PickScore, CLIPScore, and HPSv2.1:

MethodReward ShapingEvaluation Reward
AWMUniform (Baseline)1.515
AWMAdditive1.478
AWMMultiplicative (Ours)1.536
DiffusionNFTUniform (Baseline)1.506
DiffusionNFTAdditive1.538
DiffusionNFTMultiplicative (Ours)1.546

An additive bonus rewards visually different images even when their quality is low. Multiplicative shaping instead multiplies the diversity weight by the reward score, so it boosts diverse images only when their quality is high. We have included this comparison in the paper.

W3. ... The paper does not ablate ... the ridge/interpolation strength ...

Q3. Please ablate the ridge coefficient, reshaping interpolation strength, group size ...

Response: We did not run hyperparameter sweeps because each full RL run takes roughly 60 GPU hours. Instead, we use one fixed set of defaults across all experiments:

  • For group size, we use 16, following the DiffusionNFT setting in Flow-Factory.
  • For the ridge coefficient, we set it to 0.001 only for numerical stability, much like adding a small epsilon to avoid dividing by zero. For reshaping interpolation strength, we choose a reasonable default of 0.01 without tuning.

This fixed set of parameters achieves consistent gains across both RL algorithms (AWM and DiffusionNFT), both text-to-image models (SD3.5-M and FLUX.2-klein-base-4B, where our method improves DrawBench reward by 3.72% over DiffusionNFT), and six training and held-out reward models. These results show that our defaults generalize without per-setting tuning. Our evaluation is at least as thorough as in prior works on RL for flow models such as GARDO, DiffusionNFT, and AWM. We have clarified these defaults in the paper.

W3. ... or whether the quality improvements remain after explicitly constraining the original mean reward.

Q3. Does the original scalar reward remain unchanged or improve when it is explicitly constrained?

Response: We do not explicitly constrain the original scalar reward. During training, our method reweights the original scalar rewards within each prompt group so distinct images receive higher weight and redundant ones receive lower weight. At test time, all methods are compared using existing reward models with unmodified rewards. Without an explicit constraint, our method already achieves mean quality gains on DrawBench across all six training and held-out reward models of 2.62% under AWM and 3.38% under DiffusionNFT over the baseline without reweighting.

W4

W4. The claim that reward-model embeddings are a general diversity signal is stronger than the evidence. Such embeddings may encode aesthetic/style variation, semantic content, or reward-model biases, not necessarily the diversity users want ... The paper needs ... task-dependent diversity definitions.

Response: Thank you for raising this point. We agree that our original phrasing was too broad. Because our method uses reward-model embeddings, it improves diversity as measured by the reward model, which may not match a user's specific intent, and we leave task-dependent diversity definitions to future work. We have narrowed our claim in the paper and discussed this in the Limitations section.

W4. ... The paper needs targeted failure analysis ...

Response: We have added a targeted failure analysis in the appendix. We find that diversity weighting reduces visual repetition without explicitly enforcing compositional constraints such as exact counts or spatial relations. For example, our method improves storefront "Hello World" text accuracy from one baseline image to all four images and varies underwater scene pose and lighting. However, it still misses exact counts on a three-cats-and-two-dogs prompt and sometimes places a cat on the wrong side of a racket.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.