- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
GeMPO proposes a analytical framework for diffusion RL that generalizes reward reweighting beyond conventional exponential weighting, including linear, square, and negative weighting to improve policy optimization and exploration.
Critical strengths
Theoretical Insight and Generality.
The paper provides a principled and insightful theoretical framework that unifies existing reweighting methods under a general measure-matching formulation, offering deeper insights into the design of diffusion RL algorithms and broad applicability across diverse tasks.
Critical weaknesses
-
Although negative reweighting yields promising improvements in some settings, its benefits are not consistent across tasks, with standard GeMPO variants achieving stronger overall performance in certain domains. This raises questions about how broadly the proposed negative-reweighting mechanism improves exploration and whether its effectiveness depends heavily on the task and reward landscape.
-
The text-to-image experiments primarily use model-based rewards, including PickScore, HPS v2.1, and CLIPScore. It remains unclear whether the observed benefits of different reweighting strategies generalize to detector-based rewards, such as GenEval or OCR, which may exhibit different reward landscapes.
Questions for authors
Can the authors establish sufficient conditions for stable optimization with negative reweighting, and provide principled guidance for selecting the truncation parameter across different tasks?