- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1(−1)
Read review
Paper summary
MIRAGE evaluates creative text-to-image outputs, such as those made for poems or stories, by reading diffusion-transformer internal states rather than relying on standard final-stage CLIP-style similarity metrics. It pulls image-token activations from chosen transformer blocks and diffusion timesteps, feeds them into separate SAEs for each block, then computes two signals from the dominant latent’s spatial map: cross-seed centroid consistency and activation concentration. These are merged into one final score. The authors test on a set of 6,000 creative prompts across SD3, PixArt-α and FLUX. On 579 human-judged pairwise samples, MIRAGE reaches r=0.280 against human preference; for context, CLIPScore hits 0.155, VQAScore 0.154, and BLIP-2 0.244. They explicitly flag that the method works only for DiT architectures and that the overall correlation remains modest.
Critical strengths
The test setting makes sense. Poems and story prompts trip up reference-based metrics and CLIP-style similarity scores, and probing internal process signals is a legitimate research direction.
The authors do acknowledge the modest correlation and DiT-only limitation in their discussion. That restraint is welcome, yet it also reveals they already recognize their results are not strong enough to match the bolder tone set in the abstract.
Critical weaknesses
This metric is overclaimed and under-validated. The abstract and contributions frame it as a practical reference-free evaluator for creative text-to-image generation, but the supporting evidence is a weak Pearson correlation. It only narrowly outperforms standard baselines and sometimes falls behind them, and there is no experiment demonstrating downstream selection or practical utility. This is my core reason for recommending rejection.
There is also a conceptual mismatch, and the method feels incremental, packaged in a formulaic submission template. Their dominant-latent spatial statistics have no grounding in the input text, so they fail to address the stated concept-level research questions. Large sections come across as repetitive rephrasing: restating the same correlation numbers, the same limitations, and the same line about providing a complementary signal. Combined with caption and aggregation bugs, the manuscript reads more like an auto-generated draft than a polished ICLR submission.
Questions for authors
(These would need strong answers to change my rating; I am not optimistic they are quick fixes.)
- Why should the dominant SAE feature (by mean activation) correspond to prompt concepts at all? Please show text-grounded alternatives (e.g., features aligned to concept tokens) vs your argmax rule on the same human set.
- How do you score a single image in deployment? How many seeds are required, and how sensitive is r to m?
- Best-of-4 / pairwise accuracy vs human majority: does MIRAGE beat BLIP-2 in any decision task?
- Table 1 vs §4.9: equal weight or harmonic mean? Which number is in Table 2?
Thank you for the detailed rebuttal and revised manuscript. The additional baselines and SAE diagnostics address several of the reviewer’s concerns. However, the revised manuscript still reports different per-dimension correlations from those stated in the rebuttal. Furthermore, reframing MIRAGE as a spatial diagnostic clarifies its scope but does not establish why its spatial statistics reliably indicate generation quality. These remaining concerns about the metric’s validity and the consistency of its evaluation lead the reviewer to maintain the original rating.