- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper defines distribution weights using the exponential of rewards and combines them with the reference model's distribution to construct a target distribution. The goal is to reduce the loss of diversity caused by scalar reward maximization and conflicts in multi-reward optimization. The paper uses KL divergence to this target distribution to motivate its training loss. Experiments on SD3.5-M and FLUX.1-dev show improvements in both single-reward and multi-reward settings, with larger gains in the multi-reward setting.
Critical strengths
- Experiments on SD3.5-M and FLUX.1-dev show improvements in both single-reward and multi-reward settings. The gains in the multi-reward setting are larger.
- The paper is clearly written and easy to follow.
Critical weaknesses
- The target distribution, constructed by combining exponential reward weights with the reference model's distribution, can be derived as the optimal solution to expected reward maximization with KL regularization:
Its optimal distribution is:
This has the same form as the target distribution defined in this paper, with . KL-regularized versions of PPO and GRPO use surrogate losses to optimize this type of regularized reward objective. Therefore, at the level of the idealized objective, there is no fundamental distinction between "Scalar Rewards" and "Reward-Induced Distributions" when KL regularization is included. I think this raises a serious concern about the paper's theoretical motivation and claimed contribution.
- The evaluated models are relatively old. The paper does not report results on newer open-source models such as Qwen-Image or Z-Image.
Questions for authors
- Can the authors explain more fully, from a theoretical perspective, how the proposed training loss differs from KL-regularized GRPO and PPO?
- Have the authors tried experiments on more recent open-source models?