- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper looks at within-prompt diversity collapse in reinforcement-learning fine-tuning of text-to-image flow models. The proposed method reuses image embeddings already computed by a reward model and solves a regularized D-optimal design problem within each prompt group. Samples that contribute complementary embedding directions get larger weights, whereas redundant samples share less weight. These weights multiplicatively reshape scalar rewards before advantage computation, without changing the sampler or the policy-update rule. The method is evaluated with AWM and DiffusionNFT on SD3.5 Medium.
Critical strengths
- The paper is well-written and easy to follow.
- The method is lightweight and has a clean interface: it reuses embeddings already produced during reward evaluation.
Critical weaknesses
-
The diveristy score is achieved by utilizing embedding from the reward model. It may constrain the deployment for the proposed method. For example, in some rule-based reward setting, there is no entrance to obtain these embeddings unless bringing in another differentiable reward model and conducting another evaluation.
-
I wonder whether diveristy score will be influenced by other factors, e.g., noises or artifacts in the images. Considering that the integration of diveristy score is only by reweighting, it is hard to judge whether the goal of deploying reweighting really aligns with the goal of enhancing image rewards.
Questions for authors
- Please see weaknesses.
- Please provide some diversity-related metrics to report the exact change after deploying the proposed method.
- In addition, some related works that discuss process rewards or conduct similarly by reweighting techniques could also be inserted and discussed in the related works, e.g, [1,2,3].
- At this time, I lean to accept for this paper. However, some concerns remain unsolved and I will defer my final decision after rebuttal.
[1] He X, Fu S, Zhao Y, et al. Tempflow-grpo: When timing matters for grpo in flow models[C]//International Conference on Learning Representations. 2026, 2026: 132869-132895.
[2] Tong Y, Liu M, Zhao C, et al. Alleviating sparse rewards by modeling step-wise and long-term sampling effects in flow-based grpo[J]. arXiv preprint arXiv:2602.06422, 2026.
[3] Deng H, Yan K, Mao C, et al. Densegrpo: From sparse to dense reward for flow matching model alignment[J]. arXiv preprint arXiv:2601.20218, 2026.
5 comments
Response to Reviewer d4j5 (2/2): W2 and Q2–Q3
W2 & Q2
W2. I wonder whether diveristy score will be influenced by other factors, e.g., noises or artifacts in the images.
Response: Noisy or artifact-corrupted images can produce unusual embeddings, but multiplicative shaping prevents our method from favoring them. An additive bonus rewards visually different images even when their quality is low. Multiplicative shaping instead multiplies the diversity weight by the reward score, so it boosts diverse images only when their quality is high.
W2. Considering that the integration of diveristy score is only by reweighting, it is hard to judge whether the goal of deploying reweighting really aligns with the goal of enhancing image rewards.
Response: Diversity reweighting aligns with improving image rewards because it prevents early collapse onto repeated copies of a single high-scoring image during RL training. Within each prompt group, regularized D-optimal weights divide one sample's weight across near-duplicates so copies do not dominate advantage estimation, while maximizing the log-determinant expands the effective support of the conditional distribution of images given the prompt across distinct high-reward modes. To the best of our knowledge, there is no theoretical analysis in RL showing that promoting diversity improves optimization of the original reward, although many empirical works such as GARDO report such gains. Along these lines, we bring tools from diversity to flow RL and show consistent empirical gains on the original reward. At test time, all methods are compared using existing reward models with unmodified rewards. Our main reward-reshaping method improves mean DrawBench quality across all six reward models and speeds up training. On Pick-a-Pic, we set the target to each baseline's highest evaluation reward within 60 GPU hours. Training speedup is the ratio between the GPU hours the baseline needs and the GPU hours our method needs to reach this target:
| Method | DrawBench Quality Gain (%) | Training Speedup |
|---|---|---|
| AWM + Ours | +2.62 | 1.38× |
| DiffusionNFT + Ours | +3.38 | 4.32× |
We have noted this empirical scope in the Limitations section.
Q2. Please provide some diversity-related metrics to report the exact change after deploying the proposed method.
Response: Thank you for this suggestion. Our method increases conditional visual diversity over the baseline under both AWM and DiffusionNFT. Following the main evaluation setup in our paper with SD3.5-M and AWM on DrawBench, we compute each metric over the 4 images conditioned on each prompt and average the scores across all 200 prompts. PickScore diversity measures cosine distance to the nearest neighbor in the prompt group, DreamSim measures pairwise perceptual distance learned from human similarity judgments, and the Vendi score computes the effective number of distinct visual modes from DreamSim embeddings:
| Method | PickScore Diversity (×10⁻³) | DreamSim | Vendi |
|---|---|---|---|
| AWM | 6.30 | 0.2310 | 1.467 |
| AWM + GARDO | 5.91 | 0.2680 | 1.535 |
| AWM + Ours | 9.18 | 0.2955 | 1.591 |
Under DiffusionNFT, our method also increases PickScore diversity by 8.45% over the baseline, from 9.61 × 10⁻³ to 10.42 × 10⁻³. We have included these held-out diversity metrics in the paper.
Q3
Q3. In addition, some related works that discuss process rewards or conduct similarly by reweighting techniques could also be inserted and discussed in the related works, e.g, [1,2,3].
[1] He X, Fu S, Zhao Y, et al. Tempflow-grpo: When timing matters for grpo in flow models[C]//International Conference on Learning Representations. 2026, 2026: 132869-132895.
[2] Tong Y, Liu M, Zhao C, et al. Alleviating sparse rewards by modeling step-wise and long-term sampling effects in flow-based grpo[J]. arXiv preprint arXiv:2602.06422, 2026.
[3] Deng H, Yan K, Mao C, et al. Densegrpo: From sparse to dense reward for flow matching model alignment[J]. arXiv preprint arXiv:2601.20218, 2026.
Response: Thank you for pointing us to these related works. TempFlow-GRPO [1], Tong et al. [2], and DenseGRPO [3] improve temporal credit assignment along the denoising trajectory through timing-aware branching, step-wise sampling effects, and dense step-level rewards. Our method instead reweights terminal rewards across images within a prompt group to reduce visual redundancy, so it is complementary to trajectory-level credit assignment. We have cited and discussed all three works in the Related Work section.
Response to Authors
Thank you for your responses.
W1: My concern is that computing diversity-induced weights requires an embedding-model forward pass and depends on its representation quality. In rule-based settings such as the OCR task in Flow-GRPO, an additional vision encoder is needed, yet its representations may not capture task-relevant semantics.
W2: Could you clarify where the bonus reward mentioned in your response is described in the paper? While diversity matters, I remain uncertain whether reweighting without additional supervision improves performance or task alignment. As this concern partly reflects my own experience, I accept author's explanation for now. No further response on this specific issue is needed. :)
I have not found substantive revisions addressing my concerns, particularly in the limitations and related work sections. Concrete manuscript revisions—not rebuttal commitments alone—are needed. Unresolved concerns may negatively affect my final assessment. I will maintain my current rating provisionally and defer my final rating until I have considered the other reviewers’ feedback and the subsequent discussion.
Response to Reviewer d4j5 (1/2): W1 and Q1
Thank you for your positive assessment and helpful feedback. We address each point below.
W1 & Q1
Response: Thank you for raising this practical point. Rule-based rewards do not require a differentiable reward model under our method, because we compute diversity weights before advantage calculation and never backpropagate gradients through the encoder. Since pure rule-based rewards do not provide embeddings, applying our method requires one frozen forward pass from an off-the-shelf vision encoder (such as CLIP or DINOv3). With learned reward models, reusing their internal embeddings adds zero extra passes. In our separate encoder ablation on DrawBench, where diversity weights are applied directly to advantages under the same joint reward objective, different frozen encoders consistently improve overall quality over that ablation's baseline:
Because our experiments evaluate learned reward models only, we have noted the extra frozen encoder pass needed for rule-based rewards in the Limitations section.