- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper observes it remains unclear whether such gains reflect broader generation improvements or primarily a redistribution of probability toward preferred outputs. It systematically studys eight representative preference-alignment methods from three complementary perspectives: preference improvement, broader generation capability, and responsible behavior. Using Best@k evaluation under increasing sampling budgets, it confirms that alignment makes human-preferred outputs easier to sample. Overall, results suggest that preference alignment reshapes the generation distribution toward preferred outputs, but these benefits do not consistently extend to broader generation quality or responsible behavior.
Critical strengths
This paper raises an important, interesting and often overlooked question: whether gains from preference alignment represent broader capability or merely a redistribution of probability mass toward preferred outputs.
Critical weaknesses
- The detailed prompt-level analyses are centered primarily on WISE with additional GenEval. Thus, it remains unclear whether the same pattern consistently holds for the evaluation perspectives.
- This paper does not introduce a new preference-alignment method, but contributes primarily through evaluation and empirical analysis. I question whether the analytical contributions of this paper surpass those of other methods in the reward overoptimization literature.
Questions for authors
- It is insufficient for performing cross-backbone study on FLUX.1-dev with only Flow-GRPO, limiting the evidence for generalization across generators and alignment methods.
- The evaluated methods rely on substantially different supervision signals, training data, and reward functions. I recommend further analysis to support the attribution of observed effects to preference alignment.
- Why k is set to 32. Does any prior experiments or related work support this selection?
- Why unsafe-generation test are not performed on the Best@k setting?
- The main results are primarily reported as point estimates and curves without systematic confidence intervals, statistical significance tests, or cross-seed stability analyses, making it difficult to assess the reliability of some relatively small differences.
- Purely for the sake of discussion: given that six different benchmarks each employ their own distinct evaluators, why not construct a unified benchmark of one's own so as to mitigate the effects of benchmark-specific evaluator bias?
Thank you for the rebuttal. The experimental results are detailed. However, the author claim that the empirical analysis of what preference-alignment gains imply for broader generation behavior. I wonder if such a contribution meets the bar for acceptance at a top-tier venue. At the current, I tend to keep the original score.