- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that aesthetic evaluation for text-to-image generation should distinguish a criterion that is inapplicable from one that is applicable but poorly realized. It introduces a role-conditioned taxonomy spanning Photography, Artwork, and Infographic content, centered on six criteria: realism, lighting, anatomy, creativity, color harmony, and typography. Based on approximately 240K images and 840K pairwise comparisons from human and GPT-assisted annotations, the authors train AWARE, a Qwen2-VL-7B-based reward model that predicts criterion-specific quality, uncertainty, and applicability. AWARE is then used in an applicability-aware RL objective that gates and weights criterion-level rewards for each prompt group. The paper also introduces AWARE-RBench for criterion-conditioned preference/applicability evaluation and AWARE-GenBench for role-conditioned generation evaluation. On the authors' benchmarks, AWARE reaches 87.54% pairwise accuracy, and AWARE-RL improves the reported role scores of Z-Image and SenseNova-U1.5-SFT. However, the paper does not isolate the contribution of learned applicability under a matched training setup, and the main evidence relies heavily on author-constructed benchmarks and GPT-based labels/evaluation whose key statistics and human validation are not reported in the submission.
Critical strengths
-
Clear and consequential problem formulation. Separating criterion applicability from criterion quality directly addresses a real semantic failure of global aesthetic rewards. The distinction is actionable for both evaluation and training, and criterion-conflict pairs make the limitation of a single scalar score concrete.
-
Ambitious end-to-end scope. The work connects a sizable preference corpus, a unified multi-criterion reward model, two diagnostic benchmarks, and RL experiments on two different generators. The reported gains on the authors' benchmarks and the use of both Z-Image and SenseNova-U1.5-SFT suggest that the idea is worth investigating beyond one model family.
Critical weaknesses
-
The experiment does not isolate the paper's core mechanism. Table 2 compares unified applicability-aware training with role-specific experts, but those conditions differ in training-data composition and do not constitute a no-applicability control on the same mixed data and budget. The single-versus-multiple reward ablation likewise tests reward combination, not applicability. Without fixed-role, no-applicability, simple-router, and oracle-applicability baselines under matched data/compute, the observed gains could come from the 840K preference pairs, criterion-specific heads, mixed-domain training, or hand-designed role rewards rather than learned applicability. This directly undermines the title claim and the main scientific contribution.
-
The evaluation is insufficiently independent and underdocumented. AWARE is trained and primarily evaluated within the authors' own taxonomy and data-generation/annotation pipeline; GPT-5.6-sol contributes labels and is also the sole AWARE-GenBench judge, merely sampled twice. The submission omits applicability/N/A prevalence, human-versus-GPT fractions, per-criterion agreement, adjudication rates, complete split counts, source audit, human calibration of GenBench. These are explicitly deferred to camera-ready. AWARE also trails HPSv3 on HPDv3 (69.56 versus 76.13), and RL reduces GenEval for both generators (73.37 to 68.38 for Z-Image; 92.86 to 90.10 for SenseNova), so external validity cannot be inferred from the reported in-house gains.
Questions for authors
-
Under exactly matched prompts, number of samples, optimization steps, and compute, what are the results for (a) fixed role-conditioned criteria and weights without predicted applicability, (b) all six quality heads without applicability gating, (c) a simple role-label router, (d) oracle human applicability, and (e) predicted applicability? This comparison would materially determine whether the paper's central mechanism is responsible for the gains.
-
How do approximately 330K five-candidate ranking tasks produce 840K pairwise examples over 240K distinct images? Please provide the exact group count, annotator-assignment count, pair-expansion rule, filtering attrition, and prompt/source/concept split audit. The current description does not allow the scale or leakage controls to be reconstructed.
-
How well does AWARE-GenBench agree with independent blinded human judgments and with judges not used in data annotation? Please report paired uncertainty/significance for the 600 prompts and explain the substantial GenEval drops after RL. These results could change whether the claimed aesthetic gains represent general improvement or benchmark-specific optimization.