- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes E2Ada, an adaptive exploration-exploitation framework for RL fine-tuning of diffusion models, aiming at reducing diversity issues and resolving reward hacking. Specifically, E2Ada introduces an intermediate exploration reward based on structural differences between consecutive predictions of clean images. The authors also provide a theoretical analysis to relate exploration rewards to policy entropy. The proposed method is evaluated across diffusion and flow matching models with multiple reward tasks and metrics, and improvements in diversity, alignments, and efficiency are reported.
Critical strengths
Instead of judging performance merely by the training reward, the authors compare image quality and diversity at similar levels of training reward, which reflects improved generation rather than merely obtaining higher scores from the reward model. Extensive experiments conducted on multiple baselines, tasks, and metrics demonstrate the effectiveness of the proposed method.
Critical weaknesses
The proof of Theorem 1 seems unclear to me. In Appendix B.1 (lines 1059-1061), the optimal value function for the combined reward is expressed as the sum of two separately optimized value functions. This does not hold in general, since the two sub-rewards may favor different subsequent actions. This might undermine the main theoretical result, and thus either a clarification on the assumptions or a corrected proof is expected.
Also, Lemma 2.1 requires specific ordering and magnitude conditions between exploration and exploitation values, but the paper does not show that the SSIM-based reward satisfies them.
Questions for authors
Please refer to the 'Critical Weaknesses' section and address the concerns raised.
Review Submitted for the Wrong Paper
Dear Reviewer,
Thank you for your review. However, your comments appear to be intended for a different paper rather than our submission. Could you please provide the correct review for our paper?
Thank you for your time and consideration.