acceptodds
Under review as a conference paper at ICLR 2027

Scored by Its Own Reward: Evaluator-Dependent Dense-Reward Speed-Ups in Diffusion RL

Abstract

The measured advantage of a graded reward can depend on the evaluator used to define success. We compare a binary CLIP-threshold reward with a dense clipped-linear CLIP reward during DDPO fine-tuning of Stable Diffusion 1.5 on 14 compositional prompts. Two campaigns record CLIP scores; campaign B additionally scores each rollout with OWLv2 and BLIP-VQA, neither used for optimization. Hard-setting thresholds target a nominal 10% base-policy success rate. The dense reward reaches the CLIP success-rate target of 0.5 earlier by 8.45 and 7.19 epochs in the two campaigns (26.8% and 22.6% fewer epochs). In campaign B, on the 13 prompts with identifiable thresholds for all three evaluators, success changes from the initial block to the final ten-block mean from .104 to .914/.901 under CLIP, from .125 to .134/.146 under OWLv2, and from .121 to .182/.193 under BLIP-VQA (dense/binary). On these prompts, the dense-minus-binary success-rate AUC effects are +.0511, −.0101, and −.0006, respectively. The two CLIP-minus-external contrasts are +.0612 and +.0517, with simultaneous 95% confidence intervals [.0359, .0865] and [.0293, .0741]; complementary sign-flip tests give . Under these evaluators, the dense-reward speed-up is largely specific to the optimized CLIP score.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, diffusion models, reward shaping, evaluation, reward hacking, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FW5XOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper compares binary and clipped-linear CLIP rewards for DDPO fine-tuning of Stable Diffusion 1.5. Across two campaigns, dense rewards reach a CLIP-defined success target earlier; in one campaign, paired comparisons show that the dense-minus-binary trajectory advantage is substantially smaller under OWLv2 and BLIP-VQA. The contribution is a controlled empirical demonstration of evaluator-dependent reward comparisons, explicitly restricted to these automated criteria.

Critical strengths
  1. The experiments are fairly comprehensive. The paper systematically compares the evaluator used for optimization with external evaluators that are not involved in optimization, and defines evaluator-specific effects as well as paired CLIP-minus-external contrasts. This provides a useful experimental and measurement framework.
Critical weaknesses
  1. The paper does not introduce a new experimental method; it mainly compares and analyzes existing approaches, and its methodological novelty is limited.
  2. The external evaluators are not human assessments. OWLv2 has zero-inflated scores, while BLIP-VQA uses only one prompt-specific yes/no question. Neither can be directly equated with genuine compositional correctness.
  3. Training uses a rolling normalization buffer of 16 rewards to compute advantages for 64 samples. The binary-reward condition is more likely to encounter an all-zero buffer early in training, which may itself create differences in the training signal.
Questions for authors
  1. What concrete improvement does the paper propose in response to its findings? Without a corresponding intervention, the work feels more like an output of an automated-research process than a complete research contribution.
  2. The analysis seems relatively superficial. Although the paper shows that reward-induced speed-ups may depend on the evaluator, it does not investigate this finding in greater detail. For example, which task and reward characteristics lead to particular speed-up effects, and how could the findings be used to improve the reward or training process?
Reviewer_cP3zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors investigate whether the apparent benefits of dense rewards over binary rewards in diffusion reinforcement learning depend on the evaluator used to measure performance. The authors compare a binary CLIP-threshold reward with a clipped-linear dense CLIP reward when fine-tuning Stable Diffusion 1.5 using DDPO. The experiments cover 14 compositional prompts across two training campaigns. In addition to CLIP, the authors employ OWLv2 and BLIP-VQA as external evaluators that are not directly involved in reward optimization.

Critical strengths
  1. The paper is well-written and easy to follow;

  2. The investigated question: whether improvements measured by the optimized reward metric translate into improvements under alternative evaluation criteria, is important.

  3. Rather than relying solely on separate significance tests for individual evaluators, the authors directly compare evaluator-specific dense-minus-binary effects using paired prompt-level analyses.

Critical weaknesses
  1. The central claim is not fully aligned with the statistical quantity directly examined in the paper: The title of this paper emphasizes evaluator-dependent dense-reward speed-ups, while the cross-evaluator analyses primarily establish differences in the dense-minus-binary success-rate AUC effects across evaluators. AUC advantages and learning speed-ups are related but conceptually distinct quantities.

  2. The caption of Table 3 states that Appendix Table 9 reports the results for all 14 prompts. However, Appendix Table 9 only presents the continuous-score results for OWLv2 and BLIP-VQA, without including the corresponding CLIP results for the full 14-prompt setting.

  3. the broader research question concerning discrepancies between optimized reward metrics and external evaluation criteria has already been investigated in prior work. For example: [1] studied reward model overoptimization, [2] investigated reward overoptimization in diffusion models, [3] examined reward hacking using held-out evaluation metrics in diffusion RL, [4] investigated threshold-based optimization using scalar reward feedback for visual generative models. Although the direct comparison of evaluator-specific dense-minus-binary effects provides a useful empirical perspective, it remains unclear how substantially this analysis advances existing understanding beyond these prior studies.

  4. Without human evaluation or other independent validation, it is difficult to determine whether the observed evaluator disagreement reflects limitations of CLIP-based optimization, limitations of the external evaluators, or both.

[1] Gao, L., Schulman, J., & Hilton, J. (2023, July). Scaling laws for reward model overoptimization. In International conference on machine learning (pp. 10835-10866). PMLR. [2] Zhang, Z., Zhang, S., Zhan, Y., Luo, Y., Wen, Y., & Tao, D. (2024). Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases. arXiv preprint arXiv:2402.08552. [3] He, H., Ye, Y., Liu, J., Liang, J., Wang, Z., Yuan, Z., ... & Pan, L. (2025). Gardo: Reinforcing diffusion models without reward hacking. arXiv preprint arXiv:2512.24138. [4] Bai, J., Lei, Y., Shi, Q., Feng, A., Xin, Y., Zhao, Z., ... & Li, J. (2026). Threshold-Guided Optimization for Visual Generative Models. arXiv preprint arXiv:2605.04653.

Questions for authors

Please see Weaknesses

Reviewer_SZ7DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the speed-up from dense rewards in reinforcement learning fine-tuning of text-to-image diffusion models is reliable. The authors compare different reward designs using Stable Diffusion 1.5. The experiments show that the speed-up from dense rewards strongly depends on the evaluation metric used for optimization. This improvement does not necessarily indicate similar gains in generation quality under other evaluation criteria.

Critical strengths
  1. The paper highlights an important but often overlooked issue in diffusion RL. Evaluating different reward designs only with the reward model used during training may mistake improvements on a specific metric for general improvements in generation quality.
  2. The paper provides a thorough evaluation from multiple perspectives, including performance, convergence speed, continuous evaluation scores, and sensitivity analysis. These results provide strong support for the main findings.
  3. The paper uses paired confidence intervals and sign-flip tests to compare the CLIP reward gains with those measured by external evaluators. The statistical analysis is relatively rigorous.
Critical weaknesses
  1. The dense reward design is limited to a single clipped-linear CLIP reward. The paper does not evaluate more fine-grained, semantics-aware dense rewards. It remains unclear whether the findings generalize to other types of dense rewards.
  2. The experiments only use Stable Diffusion 1.5 as the base model. Evaluation on stronger diffusion models and larger, more diverse prompt sets is needed to establish the generality of the findings.
  3. OWLv2 and BLIP-VQA have their own limitations as external evaluators. It is difficult to determine whether the lack of improvement comes from the limited effectiveness of dense rewards or the limited sensitivity of these evaluators.
Questions for authors
  1. Would the observed evaluator dependence remain consistent when using more recent text-to-image models and more comprehensive benchmarks?
  2. Although the authors provide continuous-score analysis, OWLv2 and BLIP-VQA still have limitations as evaluators. Could the authors further validate the main findings through human evaluation or more established compositional evaluation benchmarks?
  3. If a more fine-grained dense reward were used during training, such as including OWLv2 scores as part of the reward, would similar evaluator dependence still be observed when using CLIP or human judgments for external evaluation?

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.