acceptodds
Under review as a conference paper at ICLR 2027

Aligned, but More Capable? Evaluating Post-Trained Text-to-Image Models Beyond Preference Metrics

Abstract

Preference alignment is widely used to improve text-to-image generation by making model outputs better match human preferences. Yet it remains unclear whether such gains reflect broader generation improvements or primarily a redistribution of probability toward preferred outputs. In this work, we systematically study eight representative preference-alignment methods from three complementary perspectives: preference improvement, broader generation capability, and responsible behavior. Using Best@ evaluation under increasing sampling budgets, we first confirm that alignment makes human-preferred outputs easier to sample. Beyond preference-oriented evaluation, however, these gains often diminish as the sampling budget increases, with the base model matching or surpassing aligned models on several benchmarks. Prompt-level analysis provides further insight: alignment does not uniformly improve success across prompts, but instead shifts the success distribution, increasing repeated-generation success for some prompts while leaving more prompts unsolved within the same budget. These changes also extend beyond capability, as most evaluated methods increase unsafe-generation rates and all increase occupational gender imbalance relative to the base model. Finally, we show that agentic generation with a frozen base model can achieve competitive generation performance while better preserving responsible behavior. Overall, our results suggest that preference alignment is better understood as reshaping the generation distribution toward preferred outputs, and motivate evaluating not only preference gains, but also how they transfer to broader capabilities and affect other model behaviors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Alignment, Text-to-image model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DnHtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes it remains unclear whether such gains reflect broader generation improvements or primarily a redistribution of probability toward preferred outputs. It systematically studys eight representative preference-alignment methods from three complementary perspectives: preference improvement, broader generation capability, and responsible behavior. Using Best@k evaluation under increasing sampling budgets, it confirms that alignment makes human-preferred outputs easier to sample. Overall, results suggest that preference alignment reshapes the generation distribution toward preferred outputs, but these benefits do not consistently extend to broader generation quality or responsible behavior.

Critical strengths

This paper raises an important, interesting and often overlooked question: whether gains from preference alignment represent broader capability or merely a redistribution of probability mass toward preferred outputs.

Critical weaknesses
  1. The detailed prompt-level analyses are centered primarily on WISE with additional GenEval. Thus, it remains unclear whether the same pattern consistently holds for the evaluation perspectives.
  2. This paper does not introduce a new preference-alignment method, but contributes primarily through evaluation and empirical analysis. I question whether the analytical contributions of this paper surpass those of other methods in the reward overoptimization literature.
Questions for authors
  1. It is insufficient for performing cross-backbone study on FLUX.1-dev with only Flow-GRPO, limiting the evidence for generalization across generators and alignment methods.
  2. The evaluated methods rely on substantially different supervision signals, training data, and reward functions. I recommend further analysis to support the attribution of observed effects to preference alignment.
  3. Why k is set to 32. Does any prior experiments or related work support this selection?
  4. Why unsafe-generation test are not performed on the Best@k setting?
  5. The main results are primarily reported as point estimates and curves without systematic confidence intervals, statistical significance tests, or cross-seed stability analyses, making it difficult to assess the reliability of some relatively small differences.
  6. Purely for the sake of discussion: given that six different benchmarks each employ their own distinct evaluators, why not construct a unified benchmark of one's own so as to mitigate the effects of benchmark-specific evaluator bias?
1 comment

Thank you for the rebuttal. The experimental results are detailed. However, the author claim that the empirical analysis of what preference-alignment gains imply for broader generation behavior. I wonder if such a contribution meets the bar for acceptance at a top-tier venue. At the current, I tend to keep the original score.

Reviewer_qn9zOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper is an empirical study like Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? But in the domain of diffusion. It asks whether the benchmark gains reported by text-to-image preference alignment reflect broader generation capability or mainly a redistribution of probability mass toward preferred outputs. Eight alignment methods (Flow-GRPO, GRPO-Guard, Diffusion-DPO, DiffusionNFT, Flow-OPD, Diffusion-OPD, RealAlign, CivitaiAlign) are evaluated on a common base model, Stable Diffusion 3.5 Medium at 512x512 (p3, p13), using Best@k under increasing sampling budgets up to k = 32 across six capability benchmarks, plus a per-prompt solvable-coverage decomposition that partitions prompts into Base-check/Post-check, Base-check/Post-cross, Base-cross/Post-check and Base-cross/Post-cross (Table 1 p6 for WISE, Table 6 p15 for GenEval). The same models are then evaluated for unsafe generation on three unsafe-prompt subsets and for occupational gender bias over 83 professions (Table 2, p7). A final section compares an agentic pipeline over the frozen base model (GEMS) and best-of-5 PickScore sampling against the aligned models (Table 3, p8), and reports one transfer check on FLUX.1-dev with Flow-GRPO (Fig. 3 and Table 4, p8). The reported conclusions are that preference gains shrink or invert as k grows, that every evaluated method loses more base-solvable prompts than it newly solves at k = 32 on WISE, and that most methods raise the unsafe ratio while all raise gender-bias MAD.

Critical strengths
  1. The solvable-transition decomposition is the right instrument for the question, and the paper applies it consistently. Reporting the four-quadrant partition per method (Table 1 p6, Table 6 p15) rather than a benchmark delta is what makes the central claim legible: Diffusion-DPO, which attains the highest Best@32 among aligned models on WISE, still gains 3.0% newly solved prompts while losing 5.9% (p6). This reframing is what the community would take away from the paper, and it is supported by numbers the authors actually report, including the cases that go against them.

  2. The study covers a genuinely current method set on one controlled base, and it reports side-effects that the alignment papers themselves do not. Eight methods spanning DPO-style, GRPO-style and curated-data alignment are run at a single resolution on SD-3.5-M (p3, p13), and the same checkpoints are then pushed through unsafe-prompt and occupational-bias protocols (Table 2, p7). The paper also flags against its own interest that the three positive Delta Solvable values on GenEval (DiffusionNFT +4.16, Flow-OPD +3.44, Diffusion-OPD +5.25) come from GenEval-oriented training rewards (p4, p15). Controlled multi-method comparisons of this kind are hard to get from individual method papers and are useful even where the interpretation is contested.

Critical weaknesses
  1. The core analysis framework appears to be a transfer of an existing LLM result that is not cited or discussed anywhere. Comparing the set of problems solvable before and after RL via pass@k, finding that the post-trained model leads at small k and is matched or overtaken at large k, and concluding that RL improves sampling efficiency rather than extending the capability boundary, is the structure of Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., arXiv:2504.13837, NeurIPS 2025), including that paper's per-problem solvable-set coverage analysis. This submission's Fig. 1, its "sampling efficiency rather than reachable quality frontier" framing (p4), and the Base-check/Post-cross quadrants (Table 1 p6, Table 6 p15) map onto it closely. This does not make the submission redundant, since the T2I setting differs in substantive ways (continuous rather than binary rewards, VLM evaluators instead of verifiers, no ground-truth verification signal), but those differences are exactly what the paper would have to argue, and right now the reader cannot tell what is new. On the T2I side the two closest works on alignment-induced distribution narrowing are cited (p2-p3) but separated by a single sentence. Positioning has to be fixed before the contribution can be assessed.

2.The general statements about preference alignment are established on one base model. The abstract and summary speak about alignment as such ("every evaluated aligned model loses more base-solvable prompts than it gains", p6; "all increase occupational gender imbalance", p1), but every one of these results is measured on SD-3.5-M at 512x512 (p3, p13). Whether alignment shrinks coverage could plausibly depend on the base: a weaker or lower-resolution base leaves more headroom for genuinely new prompts, a stronger one less. The only second base, FLUX.1-dev, is tested with Flow-GRPO alone on GenEval Best@k and unsafe ratio (Fig. 3 and Table 4, p8); the solvable-coverage analysis, which is the paper's main instrument, is not run there, and Fig. 3's own caption says the base model only "slightly" overtakes Flow-GRPO. The generalization claim of Sec. 6.2 therefore rests on one method and two metrics. Repeating the coverage analysis on FLUX.1-dev (or another 2025-era base) for at least two or three of the methods would turn these from observations about one model into findings about alignment.

Questions for authors
  1. Please cite and discuss Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., arXiv:2504.13837, NeurIPS 2025), and state explicitly what is new here beyond porting pass@k crossing and solvable-set coverage from LLM RLVR to T2I. In particular: does the continuous-valued Best@k estimator, or the fact that the evaluator is itself a VLM rather than a verifier, change the conclusion qualitatively?
  2. Could authors run the solvable-coverage analysis (the Table 1 and Table 6 partition) on FLUX.1-dev for more than Flow-GRPO. Does a net loss in coverage still appear on a different base?
  3. The p6 header says alignment "consistently increases unsafe generation" while RealAlign and CivitaiAlign reduce the average (18.93 to 18.17 and 17.18).
  4. "Reshapes the generation distribution toward preferred outputs" (p1) is a distributional claim. Can you report a direct measurement, for example per-prompt output diversity or a distance between base and aligned output distributions, instead of inferring it from Best@k and success histograms?
  5. For Table 3, what is the generation and compute budget of GEMS (including the external multi-turn model) relative to the preference-aligned models and to best-of-5? Is any part of the GEMS advantage on WISE (0.673 against 0.373) attributable to the external model's prompt rewriting rather than to the frozen generator?
1 comment

Thank authors for providing detailed response, I will keep my score.

Reviewer_L6vfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on analyzing how preference alignment changes text-to-image generation. The authors find out that alignment effectively increases the generation of preferred outputs, but not for the broader generation. So, the claim is that the preference-alignment gains should be interpreted together with their transfer to broader capabilities.

Critical strengths
  • The comparison is tightly controlled. The generators are compared with same prompts, seeds, and sampling set-up.
  • It's interesting that the generation coverage decreases, so that the non-success prompts increases, and that the capability of the model itself does not increase.
  • It's surprising that the unsafe rate increased.
Critical weaknesses
  • Most evaluated methods are trained on PickScore or its underlying preference data, and the only two with a different signal are also the only two that improve safety. The results therefore support claims about PickScore based alignment, not preference alignment in general.
  • Diversity loss and capability are not disentangled. Alignment sharpens the output distribution by design, so a higher entropy base model is expected to win at large budgets regardless of reachable capability. Without a diversity measurement or an entropy matched control, the Best@k and coverage results cannot separate reduced capability from reduced variance.
  • The GEMS result is overclaimed. It is an existing method run once with a large multimodal agent that can rewrite prompts and inject world knowledge, and compute is not matched against any baseline. The WISE gain is most naturally explained by the agent supplying knowledge the generator lacks, so it should not appear as a headline finding.
  • The central thesis is already established in the LLM literature on reinforcement learning and pass@k. The paper neither cites nor positions itself against that work, so the contribution reads as confirmation rather than discovery.
Questions for authors
  • Can you report sample diversity for base and aligned models, and does the coverage loss survive once diversity is matched (for example by varying guidance scale on the base model)?
  • Most methods share PickScore as the reward. Does the gender bias shift still appear with a different reward model such as HPSv2?
  • How does GEMS compare against a best of N baseline with the same generator budget, and does the WISE gain remain if the agent cannot rewrite the prompt?
  • Do the safety and gender evaluations align with other classifiers or evaluation methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.