- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Pairwise Reward Feedback Learning (PairReFL) to improve the reward hacking problem when applying vanilla Reward Feedback Learning (ReFL) that maximizes the reward score. Here, differentiable pairwise reward model (PairHPSv3) is applied to compare two similar images generated by current models. By backpropagating the preference signal through the better image to the generator (the other image is a detached anchor), the proposed PairReFL aim to avoid the reward hacking. Experimental results show that the proposed PairReFL outperforms ReFL, Flow-GRPO, and Pref-GRPO in terms of four preference metrics and human evaluation.
Critical strengths
- S1. The technical design is clear and reasonable. Instead of directly subtracting the reward of two images, the proposed PairReFL observes two images with the shared initial noise and stop gradient to make differentiable preference optimization.
- S2. The experiments are conducted on modern generative models, i.e., using FLUX.1-dev, together with the comparisons of ReFL and two GRPO-based methods. Table 1 shows that the proposed approach outperforms the baselines. Moreover, the analysis is comprehensive and provide some insights.
Critical weaknesses
-
W1. The motivation seems to be the problem that pointwise reward maximization is inclined to reward hacking. Nevertheless, this problem has been widely-studied before. There are many approaches that do not require explicit rewards, e.g., Diffusion-DPO (CVPR 2024), SPIN-Diffusion (NeurIPS 2024), and Diffusion-DRO (NeurIPS 2025). More importantly, the work of “Reward Sharpness-Aware Fine-Tuning” (CVPR 2026) has studied the problem of how to keep differentiable reward feedback while alleviating the reward hacking. Although this work is cited, it is not directly compared. It is suggested to justify why we need pairwise reward model instead of using reward-model-free preference optimization.
-
W2. Since PairReFL changes the reward representation and generator optimization strategy at the same time, the contribution of pairwise reward model and optimization objective is not fully separated. For instance, Table 2 compares the paired version and pointwise version, but the pairwise reward model is not always better on all the benchmark. It is suggested to provide the baseline only using pointwise HPSv3 scores as differentiable feedback.
-
W3. The qualitative examples in Figure 3 do not support the contextual reasoning and instruction fidelity well. For the prompt of “gnomes are playing music during Independence Day festivities in a forest near Lake George”, PairReFL additionally generates US flags. Although the US flags fit the context of the Independence Day, other methods generate gnomes playing music, which is also good. The US flag is not the necessary condition for the prompt. Moreover, for the prompt of “ a red-haired female knight with a golden prosthetic arm wielding a long golden blade”, PairReFL obviously changes the pose and the composition to make the long golden blade obvious. However, the result of FLUX.1-dev is also good. The golden blade, though partial, follows the prompt. It may be better alignws with “wielding”. It would be great if the authors can guide the readers to analyze the results.
-
W4 [minor]. From Appendix B.7, PairReFL shows reward-related stylistic bias. Moreover, Table 8 shows that PairReFL demonstrates greater representation drift and lower diversity retention, even the drift can provide more reward improvement. Therefore, the experiment may support the fact that PairReFL can alleviate reward overoptimization in some cases instead of solving reward hacking.
Questions for authors
- Q1. Compared with existing approaches that do not require explicit reward models, e.g., SPIN-Diffusion, Diffusion-DRO, what is the main advantage of the proposed method?
- Q2. It is suggested to provide the controlled ablation, which separates the reward representation (pointwise score difference or joint pairwise comparator) and generator optimization objective under the same computation budget.
- Q3. In Figure 3, does US flag is a better contextual reasoning? For the example of female knight, PairReFL obviously show different layouts. It is suggested to have some quantitative analysis to prove that the change improves the instruction fidelity instead of just reward score.
6 comments
W3 and Q3 (Figure 3 and instruction fidelity). We thank the reviewer for the careful reading of Figure 3. We agree that these two examples are open to interpretation, and we will remove the claim about contextual reasoning and revise the caption to guide the reader row by row. For quantitative evidence beyond reward scores, the human side-by-side study compares images generated from the same prompt and seed, and annotators prefer PairReFL over every baseline.
| Opponent | Good % | Same % | Bad % | Net (pp) |
|---|---|---|---|---|
| FLUX.1-dev | 46.0 | 26.4 | 27.6 | +18.4 |
| ReFL | 34.1 | 37.3 | 28.6 | +5.5 |
| Flow-GRPO | 39.5 | 31.8 | 28.7 | +10.8 |
| Pref-GRPO | 38.8 | 32.4 | 28.8 | +10.0 |
We also evaluated all methods on DrawBench [1], an independent set of 200 prompts designed to probe colors, counting, spatial relations, text rendering, and rare words. Only the prompt set differs from Table 1, and no method was trained on DrawBench prompts.
| Method | HPSv2.1 | HPSv3 | PickScore | ImageReward |
|---|---|---|---|---|
| FLUX.1-dev | 0.3001 | 11.7541 | 22.9025 | 0.9574 |
| ReFL | 0.3053 | 11.9040 | 22.8933 | 1.0034 |
| Flow-GRPO | 0.3023 | 11.8252 | 22.9006 | 0.9625 |
| Pref-GRPO | 0.3046 | 11.8971 | 22.9006 | 0.9878 |
| PairReFL | 0.3065 | 12.0230 | 22.9132 | 1.0148 |
PairReFL achieves the highest score on all four metrics, three of which are held out from training, and the ranking of the methods agrees with Table 1.
W4 (reward over-optimization versus reward hacking). We agree that mitigating reward over-optimization describes our contribution more precisely, and we will revise the wording throughout the paper. The training curves in Figure 2 show this effect directly. ReFL's HPSv3 rises while its held-out PickScore falls almost in mirror image, whereas PairReFL gains about 1.7 times as much HPSv3 over the same budget while its PickScore holds steady. The appearance and drift statistics point the same way once they are related to the gain they buy. Per unit of HPSv3 gain, PairReFL shows the smallest saturation and glossiness shifts of the four methods and less than half the high-frequency shift of ReFL. In Table 8, it converts drift into preference gain at 3.31 HPSv3 points per unit of DINOv2 drift against 2.27 for ReFL, the highest rate of all methods, and it loses less diversity per unit of gain than ReFL and Pref-GRPO.
[1] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in neural information processing systems 35 (2022): 36479-36494.
Kindly requesting further discussion
Dear Reviewer uB6Q,
Thank you again for your positive assessment and helpful suggestions. We would like to kindly follow up on our response. Following your comments, we added a direct comparison with Reward Sharpness-Aware Fine-Tuning under the same budget and evaluation protocol, the controlled ablation that separates the joint comparator from the HPSv3 score difference, and evidence beyond reward scores from the human study and the independent DrawBench prompt set. We will also revise the wording to "mitigating reward over-optimization" as you suggested.
We would greatly appreciate your feedback on whether these additions address your questions. If they do, we would be grateful if you could consider further strengthening your support for the paper.
Best regards, The Authors
We thank the reviewer for the positive assessment and the helpful suggestions. We address each point below.
W1 and Q1 (reward-model-free methods; Reward Sharpness-Aware Fine-Tuning). Reward-model-free methods such as Diffusion-DPO, SPIN-Diffusion, and Diffusion-DRO learn directly from the human data in a fixed dataset. SPIN-Diffusion and Diffusion-DRO also contrast the data with samples from the generator itself, but the images they move toward are still the winner or expert images of that dataset, so the generator can only learn what those images already show. A reward model learns the preference from the same kind of data and can then judge any new image, so its training signal follows what the current generator produces. Reward feedback learning further backpropagates through the reward model, which gives each fresh sample a direction of improvement rather than only a verdict. The main advantage of PairReFL over reward-model-free methods is this online, pathwise signal. PairReFL keeps it and replaces the pointwise objective, a main driver of reward over-optimization, with a pairwise comparison. PairHPSv3 is trained on the same HPDv3 preference data as HPSv3, so it requires no additional human data. Reward Sharpness-Aware Fine-Tuning, or RSA-FT, addresses reward hacking within the same differentiable-feedback paradigm by flattening the pointwise reward landscape, whereas PairReFL replaces the pointwise objective with a pairwise comparison. Because no official code is available, we reimplemented RSA-FT on ReFL following the hyperparameters reported in the paper, with HPSv3 as the reward model, the same training budget, and the evaluation protocol of Table 1.
PairReFL achieves higher HPSv3 and higher held-out PickScore than RSA-FT, while RSA-FT is comparable on HPSv2.1 and higher on ImageReward. RSA-FT does not raise HPSv3 or PickScore above ReFL, whereas PairReFL improves over ReFL on all four metrics. We will add this comparison and discussion to the revised paper.
W2 and Q2 (separating the reward representation from the optimization objective). The baseline that uses only pointwise HPSv3 scores as differentiable feedback is the ReFL row of Table 1. To separate the reward representation from the optimization objective under the same computation budget, as the reviewer suggests, we added a control. It keeps the pairwise scoring method and every training setting of PairReFL unchanged and changes only the comparator from PairHPSv3 to the HPSv3 score difference on its native preference scale. All rows use the evaluation protocol of Table 1.
The score-difference control improves over ReFL on all four metrics. With the pairwise scoring method held fixed, changing the comparator from the HPSv3 score difference to PairHPSv3 further improves HPSv2.1, HPSv3, and PickScore, and ImageReward is comparable. Both the pairwise scoring method and the joint comparator therefore contribute to the gain.
Regarding Table 2, PairHPSv3 trails HPSv3 on GenAI-Bench, whose pairs come from different generators and often differ greatly in style and content. PairReFL only asks the comparator to judge two images from the same generator, prompt, and initial latent. On held-out data closer to this setting, PairHPSv3 is more accurate than HPSv3, with 77.7% versus 76.9% on HPDv3 and 98.9% versus 97.1% on the MJ-Bench split, which isolates paired differences in blur and visual artifacts. On the shared-noise pairs encountered in training, Table 5 shows that its verdict flips under preference-preserving perturbations on 9.0% of pairs, against 13.6% for the HPSv3 score difference. The score-difference control above tests this setting directly, and PairHPSv3 trains the better generator.