acceptodds
Under review as a conference paper at ICLR 2027

PairReFL: Pairwise Reward Feedback Learning for Diffusion Models

Abstract

Aligning text-to-image models to human preferences, guided by a reward model, has become common practice. Compared with policy-gradient methods relying soly on scalar reward, reward feedback learning methods backpropagate feedback through the reward model, extracting richer training signals from each sample. Despite its efficiency, this learning paradigm is susceptible to reward hacking: maximizing a reward model's per-image score can amplify undesirable visual patterns such as oversaturated colors, without improving perceived image quality. In this paper, we propose Pairwise Reward Feedback Learning (PairReFL), which replaces scalar-score maximization with a differentiable comparison against another image sampled from the current model, so the comparison reference changes as the model learns. Across four human-preference metrics, PairReFL outperforms pointwise reward feedback learning by up to 1.1% and the strongest policy-gradient baseline by up to 3.7%. Our differentiable pairwise reward model reduces the error rate in visual-quality comparisons by 62% relative to its pointwise counterpart. Further analyses trace these gains to the pairwise interface, which provides more stable feedback on closely matched samples and achieves greater preference gains for a given degree of model change.

Primary area:
reinforcement learning
Keywords:
Text-to-Image Generation, Diffusion Models, Flow Matching, Reward Feedback Learning, Pairwise Preference, Reward Hacking, Human Preference Alignment, Reward Models, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uB6QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Pairwise Reward Feedback Learning (PairReFL) to improve the reward hacking problem when applying vanilla Reward Feedback Learning (ReFL) that maximizes the reward score. Here, differentiable pairwise reward model (PairHPSv3) is applied to compare two similar images generated by current models. By backpropagating the preference signal through the better image to the generator (the other image is a detached anchor), the proposed PairReFL aim to avoid the reward hacking. Experimental results show that the proposed PairReFL outperforms ReFL, Flow-GRPO, and Pref-GRPO in terms of four preference metrics and human evaluation.

Critical strengths
  • S1. The technical design is clear and reasonable. Instead of directly subtracting the reward of two images, the proposed PairReFL observes two images with the shared initial noise and stop gradient to make differentiable preference optimization.
  • S2. The experiments are conducted on modern generative models, i.e., using FLUX.1-dev, together with the comparisons of ReFL and two GRPO-based methods. Table 1 shows that the proposed approach outperforms the baselines. Moreover, the analysis is comprehensive and provide some insights.
Critical weaknesses
  • W1. The motivation seems to be the problem that pointwise reward maximization is inclined to reward hacking. Nevertheless, this problem has been widely-studied before. There are many approaches that do not require explicit rewards, e.g., Diffusion-DPO (CVPR 2024), SPIN-Diffusion (NeurIPS 2024), and Diffusion-DRO (NeurIPS 2025). More importantly, the work of “Reward Sharpness-Aware Fine-Tuning” (CVPR 2026) has studied the problem of how to keep differentiable reward feedback while alleviating the reward hacking. Although this work is cited, it is not directly compared. It is suggested to justify why we need pairwise reward model instead of using reward-model-free preference optimization.

  • W2. Since PairReFL changes the reward representation and generator optimization strategy at the same time, the contribution of pairwise reward model and optimization objective is not fully separated. For instance, Table 2 compares the paired version and pointwise version, but the pairwise reward model is not always better on all the benchmark. It is suggested to provide the baseline only using pointwise HPSv3 scores as differentiable feedback.

  • W3. The qualitative examples in Figure 3 do not support the contextual reasoning and instruction fidelity well. For the prompt of “gnomes are playing music during Independence Day festivities in a forest near Lake George”, PairReFL additionally generates US flags. Although the US flags fit the context of the Independence Day, other methods generate gnomes playing music, which is also good. The US flag is not the necessary condition for the prompt. Moreover, for the prompt of “ a red-haired female knight with a golden prosthetic arm wielding a long golden blade”, PairReFL obviously changes the pose and the composition to make the long golden blade obvious. However, the result of FLUX.1-dev is also good. The golden blade, though partial, follows the prompt. It may be better alignws with “wielding”. It would be great if the authors can guide the readers to analyze the results.

  • W4 [minor]. From Appendix B.7, PairReFL shows reward-related stylistic bias. Moreover, Table 8 shows that PairReFL demonstrates greater representation drift and lower diversity retention, even the drift can provide more reward improvement. Therefore, the experiment may support the fact that PairReFL can alleviate reward overoptimization in some cases instead of solving reward hacking.

Questions for authors
  • Q1. Compared with existing approaches that do not require explicit reward models, e.g., SPIN-Diffusion, Diffusion-DRO, what is the main advantage of the proposed method?
  • Q2. It is suggested to provide the controlled ablation, which separates the reward representation (pointwise score difference or joint pairwise comparator) and generator optimization objective under the same computation budget.
  • Q3. In Figure 3, does US flag is a better contextual reasoning? For the example of female knight, PairReFL obviously show different layouts. It is suggested to have some quantitative analysis to prove that the change improves the instruction fidelity instead of just reward score.
6 comments

We thank the reviewer for the positive assessment and the helpful suggestions. We address each point below.

W1 and Q1 (reward-model-free methods; Reward Sharpness-Aware Fine-Tuning). Reward-model-free methods such as Diffusion-DPO, SPIN-Diffusion, and Diffusion-DRO learn directly from the human data in a fixed dataset. SPIN-Diffusion and Diffusion-DRO also contrast the data with samples from the generator itself, but the images they move toward are still the winner or expert images of that dataset, so the generator can only learn what those images already show. A reward model learns the preference from the same kind of data and can then judge any new image, so its training signal follows what the current generator produces. Reward feedback learning further backpropagates through the reward model, which gives each fresh sample a direction of improvement rather than only a verdict. The main advantage of PairReFL over reward-model-free methods is this online, pathwise signal. PairReFL keeps it and replaces the pointwise objective, a main driver of reward over-optimization, with a pairwise comparison. PairHPSv3 is trained on the same HPDv3 preference data as HPSv3, so it requires no additional human data. Reward Sharpness-Aware Fine-Tuning, or RSA-FT, addresses reward hacking within the same differentiable-feedback paradigm by flattening the pointwise reward landscape, whereas PairReFL replaces the pointwise objective with a pairwise comparison. Because no official code is available, we reimplemented RSA-FT on ReFL following the hyperparameters reported in the paper, with HPSv3 as the reward model, the same training budget, and the evaluation protocol of Table 1.

MethodHPSv2.1HPSv3PickScoreImageReward
ReFL0.315413.819622.76421.1138
ReFL + RSA-FT0.318313.813322.74011.1515
PairReFL0.318213.967722.83481.1252

PairReFL achieves higher HPSv3 and higher held-out PickScore than RSA-FT, while RSA-FT is comparable on HPSv2.1 and higher on ImageReward. RSA-FT does not raise HPSv3 or PickScore above ReFL, whereas PairReFL improves over ReFL on all four metrics. We will add this comparison and discussion to the revised paper.

W2 and Q2 (separating the reward representation from the optimization objective). The baseline that uses only pointwise HPSv3 scores as differentiable feedback is the ReFL row of Table 1. To separate the reward representation from the optimization objective under the same computation budget, as the reviewer suggests, we added a control. It keeps the pairwise scoring method and every training setting of PairReFL unchanged and changes only the comparator from PairHPSv3 to the HPSv3 score difference on its native preference scale. All rows use the evaluation protocol of Table 1.

MethodHPSv2.1HPSv3PickScoreImageReward
ReFL (pointwise HPSv3)0.315413.819622.76421.1138
PairReFL procedure + HPSv3 score difference0.316513.867422.80221.1259
PairReFL (PairHPSv3)0.318213.967722.83481.1252

The score-difference control improves over ReFL on all four metrics. With the pairwise scoring method held fixed, changing the comparator from the HPSv3 score difference to PairHPSv3 further improves HPSv2.1, HPSv3, and PickScore, and ImageReward is comparable. Both the pairwise scoring method and the joint comparator therefore contribute to the gain.

Regarding Table 2, PairHPSv3 trails HPSv3 on GenAI-Bench, whose pairs come from different generators and often differ greatly in style and content. PairReFL only asks the comparator to judge two images from the same generator, prompt, and initial latent. On held-out data closer to this setting, PairHPSv3 is more accurate than HPSv3, with 77.7% versus 76.9% on HPDv3 and 98.9% versus 97.1% on the MJ-Bench split, which isolates paired differences in blur and visual artifacts. On the shared-noise pairs encountered in training, Table 5 shows that its verdict flips under preference-preserving perturbations on 9.0% of pairs, against 13.6% for the HPSv3 score difference. The score-difference control above tests this setting directly, and PairHPSv3 trains the better generator.

W3 and Q3 (Figure 3 and instruction fidelity). We thank the reviewer for the careful reading of Figure 3. We agree that these two examples are open to interpretation, and we will remove the claim about contextual reasoning and revise the caption to guide the reader row by row. For quantitative evidence beyond reward scores, the human side-by-side study compares images generated from the same prompt and seed, and annotators prefer PairReFL over every baseline.

OpponentGood %Same %Bad %Net (pp)
FLUX.1-dev46.026.427.6+18.4
ReFL34.137.328.6+5.5
Flow-GRPO39.531.828.7+10.8
Pref-GRPO38.832.428.8+10.0

We also evaluated all methods on DrawBench [1], an independent set of 200 prompts designed to probe colors, counting, spatial relations, text rendering, and rare words. Only the prompt set differs from Table 1, and no method was trained on DrawBench prompts.

MethodHPSv2.1HPSv3PickScoreImageReward
FLUX.1-dev0.300111.754122.90250.9574
ReFL0.305311.904022.89331.0034
Flow-GRPO0.302311.825222.90060.9625
Pref-GRPO0.304611.897122.90060.9878
PairReFL0.306512.023022.91321.0148

PairReFL achieves the highest score on all four metrics, three of which are held out from training, and the ranking of the methods agrees with Table 1.

W4 (reward over-optimization versus reward hacking). We agree that mitigating reward over-optimization describes our contribution more precisely, and we will revise the wording throughout the paper. The training curves in Figure 2 show this effect directly. ReFL's HPSv3 rises while its held-out PickScore falls almost in mirror image, whereas PairReFL gains about 1.7 times as much HPSv3 over the same budget while its PickScore holds steady. The appearance and drift statistics point the same way once they are related to the gain they buy. Per unit of HPSv3 gain, PairReFL shows the smallest saturation and glossiness shifts of the four methods and less than half the high-frequency shift of ReFL. In Table 8, it converts drift into preference gain at 3.31 HPSv3 points per unit of DINOv2 drift against 2.27 for ReFL, the highest rate of all methods, and it loses less diversity per unit of gain than ReFL and Pref-GRPO.

[1] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in neural information processing systems 35 (2022): 36479-36494.

AuthorsOpenReview ↗

Kindly requesting further discussion

Dear Reviewer uB6Q,

Thank you again for your positive assessment and helpful suggestions. We would like to kindly follow up on our response. Following your comments, we added a direct comparison with Reward Sharpness-Aware Fine-Tuning under the same budget and evaluation protocol, the controlled ablation that separates the joint comparator from the HPSv3 score difference, and evidence beyond reward scores from the human study and the independent DrawBench prompt set. We will also revise the wording to "mitigating reward over-optimization" as you suggested.

We would greatly appreciate your feedback on whether these additions address your questions. If they do, we would be grateful if you could consider further strengthening your support for the paper.

Best regards, The Authors

Reviewer_L6KsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PairReFL keeps the gradient pathway of reward feedback learning (backpropagating a differentiable reward through generation) but replaces the pointwise score with a differentiable pairwise comparator, PairHPSv3, a Qwen2-VL-7B fine-tuned on HPDv3 to answer whether the second image beats the first. For each prompt, two images are produced from a shared initial latent with independent stochastic perturbations; the comparator is evaluated twice with one image detached each time, and a hinge-capped preference loss is backpropagated into the image receiving the gradient. On FLUX.1-dev with HPDv2 prompts, PairReFL outperforms ReFL, Flow-GRPO and Pref-GRPO on four automatic metrics, supported by a human GSB study, prompt-paired bootstrap intervals, training-dynamics plots that expose reward hacking in ReFL, drift/diversity analyses, and diagnostics on comparator antisymmetry and the necessity of the stop-gradient.

Critical strengths
  1. Joint pairwise reward backpropagation is a clear and practically relevant extension of direct reward tuning. The stop-gradient analysis and peer-dependent gradient measurements make the mechanism concrete.
  2. The evaluation includes human judgments, prompt-paired bootstrap intervals, and extensive drift, diversity, and appearance diagnostics. These provide substantially stronger support than automated reward scores alone.
Critical weaknesses
  1. The central reward-hacking explanation is stronger than the evidence supports. Removing absolute scores does not itself remove exploitation: with a score-difference comparator, detaching the peer still produces positively weighted ascent on each image’s scalar score. Appendix B.10 establishes peer dependence for the actual comparator, but not that its gradients avoid undesirable shared attributes. Indeed, Appendix B.7 reports appearance shifts comparable to ReFL and a decline in medium fidelity from 0.5283 to 0.4811. These findings do not negate the preference gains, but leave the proposed mechanism of reduced reward hacking insufficiently established.
  2. Training ablations do not isolate the source of improvement. The method simultaneously introduces a jointly trained comparator, shared-noise pairing, bilateral stop-gradient, and a confidence cap. Table 3 tests peer source and initial-noise sharing, but does not replace PairHPSv3 with an HPSv3 score difference while preserving the same rollout, cap, and update rule. Nor is the cap’s benefit established through retraining.
  3. The evidence for practical superiority remains narrow. Generator evaluation uses one backbone and one prompt benchmark. The comparator itself drops from HPSv3’s 72.7% to 69.4% on GenAI-Bench, making transfer particularly relevant. The human study is encouraging, but the ReFL comparison contains only 125 distinct pairs, and its 5.5-point net advantage lacks pair-aware uncertainty.
Questions for authors
  1. What happens when the joint comparator is replaced by an HPSv3 score difference under an otherwise identical PairReFL training recipe?
  2. Does the advantage persist across training seeds and at matched GPU-hours, including reward-model costs?
  3. Can the authors report pair-aware uncertainty for the human comparison against ReFL and evaluate prompt-specific fidelity on an independent benchmark?
8 comments

We thank the reviewer for the detailed and precise review. We address each point below.

W1 (strength of the reward-hacking explanation). We agree that our evidence supports mitigating reward over-optimization rather than eliminating reward hacking, and we will revise the wording throughout the paper. Our results show that PairReFL exploits reward-hacking features less than ReFL at a common reward gain. For each method, the table below places the HPSv3 gain over FLUX.1-dev reported in Table 1 against the shift of three Appendix B.7 statistics, namely saturation, glossiness measured along the CLIP glossy axis, and high-frequency energy. The value in parentheses divides each shift by the HPSv3 gain.

MethodHPSv3 gainSaturation shiftGlossy-axis shiftHigh-frequency shift
Flow-GRPO+0.1340.0183 (0.137)0.0035 (0.026)0.0001 (0.001)
Pref-GRPO+0.2200.0355 (0.161)0.0074 (0.034)0.0022 (0.010)
ReFL+0.2620.0301 (0.115)0.0101 (0.039)0.0111 (0.042)
PairReFL+0.4100.0351 (0.086)0.0081 (0.020)0.0078 (0.019)

In absolute terms, the shifts of PairReFL are of the same order as those of ReFL and Pref-GRPO, as Appendix B.7 reports, while its reward gain is the largest, about 1.6 times that of ReFL. Per unit of reward gain, PairReFL shifts saturation and glossiness the least of all four methods and shifts high-frequency energy less than half as much as ReFL. Flow-GRPO and Pref-GRPO shift high-frequency energy less per unit of gain, but they also obtain the two smallest reward gains. We also extended the medium-fidelity analysis of Appendix B.7 to all baselines.

MethodMedium fidelity
FLUX.1-dev0.528
ReFL0.462
Flow-GRPO0.528
Pref-GRPO0.514
PairReFL0.481

Medium fidelity declines less for PairReFL than for ReFL, and the two policy-gradient methods, which gain less reward, stay closer to FLUX.1-dev.

We agree with the reviewer's point on the mechanism. For the HPSv3 score difference, which subtracts two independently computed HPSv3 scores, the gradient with respect to the active image is the gradient of its own HPSv3 score, so the pairwise scoring method alone does not remove exploitation. It only weights this ascent by the comparison with the shared-noise peer and stops it once the confidence cap is reached. The joint comparator conditions on both images, so its gradient for the active image changes with the peer. Appendix B.10 shows that swapping the detached peer changes the direction of this gradient by a cosine distance of 0.172, whereas the HPSv3 score difference gives zero by construction. Peer dependence alone does not show that shared undesirable attributes are avoided, so we tested the comparator directly with the score-difference control described under W2 and Q1, which changes only the comparator from PairHPSv3 to the HPSv3 score difference. On HPSv2.1, HPSv3, and PickScore, the control falls between ReFL and PairReFL, so the joint comparator contributes beyond the pairwise scoring method.

W2 and Q1 (isolating the comparator and retraining the confidence cap). We thank the reviewer for proposing this control. As required, we have added the experiment, which keeps the pairwise scoring method and every training setting of PairReFL unchanged and changes only the comparator from PairHPSv3 to the HPSv3 score difference on its native preference scale. All rows use the evaluation protocol of Table 1.

MethodHPSv2.1HPSv3PickScoreImageReward
ReFL (pointwise HPSv3)0.315413.819622.76421.1138
PairReFL procedure + HPSv3 score difference0.316513.867422.80221.1259
PairReFL (PairHPSv3)0.318213.967722.83481.1252

The score-difference control improves over ReFL on all four metrics. With the pairwise scoring method held fixed, changing the comparator from the HPSv3 score difference to PairHPSv3 further improves HPSv2.1, HPSv3, and PickScore, and ImageReward is comparable. Both the pairwise scoring method and the joint comparator therefore contribute to the gain, and PairReFL combines them as complementary components.

We also retrained PairReFL without the confidence cap (τ = 1.0), keeping everything else unchanged.

SettingHPSv2.1HPSv3PickScoreImageReward
PairReFL, τ = 0.9 (default)0.318213.967722.83481.1252
PairReFL, τ = 1.0 (no cap)0.318513.947222.79771.1348

Without the confidence cap, HPSv2.1 and ImageReward are comparable, and HPSv3 and PickScore are slightly lower. The confidence cap therefore brings a small gain on the in-domain and held-out preference scores, and PairReFL stays ahead of ReFL on all four metrics without it.

W3 and Q3 (one backbone and one prompt benchmark, comparator transfer, and human-study uncertainty). To test whether the gains transfer to a second generator backbone, we trained ReFL and PairReFL on SD3.5-Medium with the configuration of the main experiments and evaluated them with the protocol of Table 1.

MethodHPSv2.1HPSv3PickScoreImageReward
SD3.5-Medium0.296311.693822.67031.0736
ReFL0.325313.363222.75161.2616
PairReFL0.337413.899322.96451.3165

PairReFL improves over ReFL on all four metrics, with larger gaps than on FLUX.1-dev. Relative to SD3.5-Medium, PairReFL gains 2.21 HPSv3 points against 1.67 for ReFL, and 0.29 PickScore points against 0.08. We will add these results to the revised paper.

To test whether the gains transfer beyond the HPDv2 prompt distribution, we repeated the evaluation of Table 1 on DrawBench [1], an independent set of 200 prompts covering colors, counting, spatial relations, text rendering, and rare words. Only the prompt set changes. The checkpoints, metrics, and four images per prompt are those of Table 1, and no method was trained on DrawBench prompts.

MethodHPSv2.1HPSv3PickScoreImageReward
FLUX.1-dev0.300111.754122.90250.9574
ReFL0.305311.904022.89331.0034
Flow-GRPO0.302311.825222.90060.9625
Pref-GRPO0.304611.897122.90060.9878
PairReFL0.306512.023022.91321.0148

PairReFL achieves the highest score on all four metrics, and the ranking of the methods agrees with Table 1. Regarding the comparator's GenAI-Bench accuracy, GenAI-Bench pairs images from different generators, often with large differences in style and content, whereas PairReFL only asks the comparator to judge two images from the same generator, prompt, and initial latent. On held-out data closer to this setting, PairHPSv3 is more accurate than HPSv3, with 77.7% versus 76.9% on HPDv3 and 98.9% versus 97.1% on MJ-Bench. On the shared-noise pairs encountered in training, Table 5 shows that its verdict flips under preference-preserving perturbations on 9.0% of pairs, against 13.6% for the HPSv3 score difference. The generator trained with PairHPSv3 also outperforms the one trained with HPSv3 on the held-out metrics, on both HPDv2 and DrawBench.

For the human study, we computed 95% confidence intervals by resampling image pairs with 10,000 bootstrap draws, so that the multiple votes on each pair are not treated as independent. Good means that annotators preferred PairReFL, Bad means that they preferred the opponent, and Net is Good minus Bad.

OpponentGood %Same %Bad %Net (pp)95% CI
FLUX.1-dev46.026.427.6+18.4[13.0, 23.9]
ReFL34.137.328.6+5.5[2.0, 9.9]
Flow-GRPO39.531.828.7+10.8[6.0, 15.5]
Pref-GRPO38.832.428.8+10.0[5.8, 14.2]

The intervals against all four opponents exclude zero, including the comparison with ReFL, which is based on 125 pairs. This agrees with our automatic evaluations, where PairReFL is ahead of ReFL on HPDv2 and on all four metrics on the independent DrawBench prompt set. We will add the DrawBench table and these intervals to the revised paper.

[1] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in neural information processing systems 35 (2022): 36479-36494.

Q2 (training seeds and matched GPU-hours). We trained PairReFL and ReFL with a second training seed under the setting of Table 1.

MethodSeedHPSv2.1HPSv3PickScoreImageReward
ReFL420.315413.819622.76421.1138
ReFL430.316713.770322.69121.1439
PairReFL420.318213.967722.83481.1252
PairReFL430.318513.957022.81991.1398

For both seeds, PairReFL is higher than ReFL on HPSv2.1, HPSv3, and PickScore, and the two PairReFL seeds agree closely. ImageReward differences between the methods are within the variation across seeds.

For matched GPU-hours, ReFL and PairReFL share the same rollout, differentiable suffix, and update, and differ only in the reward model. The joint comparator scores each image together with its peer, so its forward and backward passes cost about twice those of a pointwise reward model. We profiled PairReFL on one node of 8 H200 GPUs. An optimizer step takes 186 s, of which the reward forward and backward passes take 13.7 s. A pointwise reward model would take about half of this time, so a ReFL step takes about 179 s and a PairReFL step costs about 1.04 times as much. The GPU-hours of 1,000 ReFL steps therefore cover about 963 PairReFL steps, and we evaluated PairReFL at step 900, which stays within this budget.

MethodStepsHPSv2.1HPSv3PickScoreImageReward
ReFL1,0000.315413.819622.76421.1138
PairReFL9000.318113.938922.82001.1353

At matched GPU-hours including reward-model costs, PairReFL remains higher than ReFL on all four metrics.

Reviewer_171TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PairReFL for text-to-image preference alignment, aiming to mitigate reward hacking caused by over-optimizing low-level visual patterns in pointwise ReFL. It i) trains a differentiable pairwise reward model and ii) uses it to compare images sampled from the current generator and directly backpropagate preference feedback. Experiments on FLUX.1-dev show improvements over pointwise ReFL and policy-gradient baselines in both automatic preference metrics and human evaluations.

Critical strengths
  1. Clear motivation. The paper addresses reward hacking in pointwise ReFL, a practical issue in preference alignment. Figure 2 clearly illustrates the divergence between in-domain reward and held-out preference scores, making the motivation easy to follow.
  2. Clear mathematical formulation. The method is clearly formulated, and the gradient-cancellation analysis under an antisymmetric comparator provides a concrete rationale for the bilateral stop-gradient design.
Critical weaknesses

Major

  1. My understanding is that pairwise comparison is intended to reduce reward hacking because undesirable visual patterns shared by both images cannot confer a preference advantage. However, with the peer held fixed, further reinforcing these patterns could still increase the active image’s preference score if the comparator remains biased toward them. More direct evidence is needed to show that PairReFL updates actually reduce the exploitation of such features.
  2. The paper emphasizes that jointly observing two images provides more effective feedback than scoring them independently. However, PairReFL changes both the reward model and the optimization objective relative to ReFL, leaving the source of improvement unclear. A key missing training control is to replace PairHPSv3 with HPSv3 score differences while keeping the paired sampling, loss, and stop-gradient scheme identical, thereby isolating the joint comparator’s additional contribution.

Minor

  1. The Reward Models discussion could include diffusion-based latent reward modeling [1, 2] to provide a more complete overview of alternative reward modeling approaches.

[1] Liu, Gongye, et al. “Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling.” arXiv preprint arXiv:2602.11146 (2026). [2] Zhang, Tao, et al. “Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization.” Advances in Neural Information Processing Systems 38 (2025).

Questions for authors
  1. Could the authors provide experiments with additional diffusion and reward model backbones? In particular, does PairReFL retain its advantage when a stronger pointwise reward model is used?
  2. How sensitive is the method to the truncation point K? Since K affects both the stochastic prefix and the differentiable suffix, could the authors ablate its sampling range and examine the effects on pair similarity and final performance?
  3. Could Fig. 5 include a plot of low-level appearance shift versus reward gain, using the color, glossiness, or frequency statistics from Appendix B.7? This would more directly show whether PairReFL produces smaller appearance shifts at comparable reward gains.
7 comments

We thank the reviewer for the careful reading and constructive suggestions. We have uploaded a separate supplementary material PDF, available under Supplementary Material on the submission page, which collects all results added during the discussion in Sections S1 to S5, with text added to the paper highlighted in blue. We address each point below.

W1 (exploitation with a fixed peer). Our results show that PairReFL exploits reward-hacking features less than ReFL, and that this comes from the joint comparator rather than from the fixed peer alone.

  • Smaller appearance shift per unit of reward gain. We divided the shift of each Appendix B.7 statistic by the HPSv3 gain over FLUX.1-dev. Per HPSv3 point, PairReFL shifts saturation by 0.086 (ReFL 0.115), the CLIP glossy axis by 0.020 (ReFL 0.039), and high-frequency energy by 0.019 (ReFL 0.042). Its saturation and glossiness ratios are the lowest of all four methods.

  • The joint comparator provides the protection. Keeping the pairwise scoring method and changing only the comparator from PairHPSv3 to the HPSv3 score difference lowers HPSv3 from 13.9677 to 13.8674 and PickScore from 22.8348 to 22.8022 (full table under W2).

  • Why a shared feature earns no advantage. With the HPSv3 score difference, the gradient for the active image is the gradient of its own HPSv3 score and does not depend on the peer. A reward-hacking feature such as a plastic-like gloss therefore raises the reward of the active image by the same amount whether or not the peer already shows it. The joint comparator conditions on both images, so its gradient for the active image changes with the peer, as shown in Appendix B.10. Because it is trained to predict which of two images people prefer, it gives little advantage to a feature that both images share.

W2 (isolating the joint comparator). We thank the reviewer for proposing this control. As required, we have added the experiment, which keeps the pairwise scoring method and every training setting of PairReFL unchanged and changes only the comparator from PairHPSv3 to the HPSv3 score difference on its native preference scale. All rows use the evaluation protocol of Table 1.

MethodHPSv2.1HPSv3PickScoreImageReward
ReFL (pointwise HPSv3)0.315413.819622.76421.1138
PairReFL procedure + HPSv3 score difference0.316513.867422.80221.1259
PairReFL (PairHPSv3)0.318213.967722.83481.1252

With the pairwise scoring method held fixed, changing the comparator from the HPSv3 score difference to PairHPSv3 improves overall performance. The score-difference control itself also improves over ReFL on all four metrics. Pairwise scoring and the joint comparator therefore each contribute to the gain of PairReFL. We have added this control to Section S2 of the supplementary PDF.

Minor (diffusion-native latent reward models). Thank you for pointing us to these works. We have added both to the Reward Models paragraph with the following discussion (Section S1 of the supplementary PDF, highlighted in blue): "Diffusion-native latent reward models [1, 2] score a noisy latent or a single denoising step inside the diffusion model, which avoids decoding and enables step-level feedback, yet they still return a pointwise score." PairHPSv3 differs in the interface rather than the representation space, because it conditions jointly on the prompt and two images and returns a differentiable preference logit.

Q1 (other backbones and a stronger pointwise reward model).

  • Second backbone. We trained ReFL and PairReFL on SD3.5-Medium with the configuration of the main experiments and evaluated them with the protocol of Table 1. The results are in Section S3 of the supplementary PDF.
MethodHPSv2.1HPSv3PickScoreImageReward
SD3.5-Medium0.296311.693822.67031.0736
ReFL0.325313.363222.75161.2616
PairReFL0.337413.899322.96451.3165

PairReFL improves over ReFL on all four metrics, with larger gaps than on FLUX.1-dev. Relative to SD3.5-Medium, PairReFL gains 2.21 HPSv3 points against 1.67 for ReFL, and 0.29 PickScore points against 0.08.

  • Stronger pointwise reward model. Our claim is a controlled one. With the same preference data and backbone, training a pairwise comparator and optimizing it with PairReFL gives larger gains than training a pointwise scorer and optimizing it with ReFL. HPSv3 is a state-of-the-art pointwise reward model, and PairHPSv3 shares its backbone and training data, which is the setting that separates the reward interface from the training data. A pointwise model trained on different or larger data would change both at once, and a controlled comparison would require a pairwise counterpart trained on the same data.

Q2 (truncation point K). We measured both quantities the reviewer asked about and added them to Section S4 of the supplementary PDF.

  • Pair similarity. We replayed the training rollout of FLUX.1-dev on 800 shared-noise pairs drawn from 400 HPDv2 test prompts with two pairs each and computed the DINOv2 cosine distance between the two images of each pair, grouped by the sampled truncation step K out of 10.
K6789
Pair distance0.4320.4870.4640.511

The distance generally grows as the truncation point moves later, since more stochastic prefix steps let the two images of a pair diverge. Averaged over the steps each range covers, the distance is 0.474 for the default range [0.625, 1.0] and 0.487 for [0.75, 1.0], so moving the range changes pair similarity only slightly.

  • Final performance. We retrained PairReFL with the differentiable suffix starting from [0.75, 1.0] instead of [0.625, 1.0], keeping everything else unchanged.
Truncation rangeHPSv2.1HPSv3PickScoreImageReward
FLUX.1-dev (no training)0.307913.557722.78741.0441
[0.625, 1.0] (default)0.318213.967722.83481.1252
[0.75, 1.0]0.321114.024422.75851.1531

A later start raises HPSv2.1, HPSv3, and ImageReward further but lowers PickScore below FLUX.1-dev, which is the held-out pattern that Table 1 shows for ReFL. The default range is the one under which PairReFL improves over FLUX.1-dev on all four metrics, which is why we keep it. Under both ranges PairReFL stays above ReFL on HPSv2.1, HPSv3, and ImageReward.

Q3 (appearance shift versus reward gain). We made the plot the reviewer suggested. It is Figure S1 in Section S5 of the supplementary PDF, and the table below gives its data.

  • What the plot shows. For each method, the plot places the HPSv3 gain over FLUX.1-dev reported in Table 1 against the shift of three Appendix B.7 statistics, namely saturation, glossiness measured along the CLIP glossy axis, and high-frequency energy. A dotted line through the origin and ReFL marks the shift of ReFL per unit of reward gain. The value in parentheses divides each shift by the HPSv3 gain, which compares the methods at a common reward gain.
MethodHPSv3 gainSaturation shiftGlossy-axis shiftHigh-frequency shift
Flow-GRPO+0.1340.0183 (0.137)0.0035 (0.026)0.0001 (0.001)
Pref-GRPO+0.2200.0355 (0.161)0.0074 (0.034)0.0022 (0.010)
ReFL+0.2620.0301 (0.115)0.0101 (0.039)0.0111 (0.042)
PairReFL+0.4100.0351 (0.086)0.0081 (0.020)0.0078 (0.019)
  • Result. PairReFL lies below the ReFL line in all three panels. In absolute terms, its shifts are of the same order as those of ReFL and Pref-GRPO, as Appendix B.7 reports, while its reward gain is the largest, about 1.6x that of ReFL. Per unit of reward gain, PairReFL shifts saturation and glossiness the least of all four methods and shifts high-frequency energy less than half as much as ReFL. PairReFL therefore obtains more reward with less of the saturated, glossy, high-frequency rendering associated with reward hacking.
AuthorsOpenReview ↗

Kindly requesting further discussion

Dear Reviewer 171T,

Thank you again for your thoughtful review. We would like to kindly follow up on our response. Following your suggestions, we added the score-difference control that changes only the comparator from PairHPSv3 to the HPSv3 score difference, a second backbone (SD3.5-Medium, where PairReFL improves over ReFL on all four metrics), the truncation-range ablation with pair similarity, and the appearance-shift versus reward-gain plot you proposed. All of them are collected in the supplementary PDF.

We would greatly appreciate it if you could let us know whether these results address your concerns. If any question remains, we are happy to clarify it during the discussion period. If you find the concerns resolved, we would be grateful if you could consider updating your assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.