acceptodds
Under review as a conference paper at ICLR 2027

Where Do Pixels Hide Their Secrets? Uncovering Model Internals for Creative Text-to-Image Evaluation

Abstract

Text-to-image (T2I) models can produce visually convincing images while still failing to correctly realize the concepts and relationships expressed in a prompt. This challenge is particularly difficult for creative inputs such as poems and sto- ries, for which reference images are typically unavailable. Existing reference-free metrics primarily evaluate the relationship between the input text and the final gen- erated image, while recent interpretability studies suggest that intermediate repre- sentations contain information about the spatial organization of generated content during the denoising process. We ask whether such model-internal signals can instead be used to evaluate generation quality. We introduce MIRAGE (Model- Internal Representation Analysis for Generation Evaluation), a reference-free metric that evaluates spatial properties of internal representations during image generation. MIRAGE extracts image-token representations from intermediate diffusion-transformer blocks and decomposes them into sparse latent features. For each generation state, it identifies the dominant sparse feature and derives two complementary spatial signals: cross-seed spatial consistency, which mea- sures the stability of the feature’s spatial centroid across stochastic generations, and activation concentration, which measures how spatially localized the feature’s activation is around its centroid. These signals are aggregated into an interpretable score for each generated image. We evaluate MIRAGE on a curated collection of 6,000 poems and stories generated using three diffusion-transformer T2I models. Across human evaluations of semantic fidelity, attribute presence, compositional correctness, and narrative fidelity, MIRAGE shows agreement with human judg- ments and provides a complementary signal to existing reference-free evaluation metrics. These results suggest that model-internal spatial representations can pro- vide useful signals for reference-free evaluation of creative T2I generation.

Primary area:
interpretability and explainable AI
Keywords:
Text-to-image generation, poem-to-image generation, metric, reference-free evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SPeMOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1(−1)
Read review
Paper summary

MIRAGE evaluates creative text-to-image outputs, such as those made for poems or stories, by reading diffusion-transformer internal states rather than relying on standard final-stage CLIP-style similarity metrics. It pulls image-token activations from chosen transformer blocks and diffusion timesteps, feeds them into separate SAEs for each block, then computes two signals from the dominant latent’s spatial map: cross-seed centroid consistency and activation concentration. These are merged into one final score. The authors test on a set of 6,000 creative prompts across SD3, PixArt-α and FLUX. On 579 human-judged pairwise samples, MIRAGE reaches r=0.280 against human preference; for context, CLIPScore hits 0.155, VQAScore 0.154, and BLIP-2 0.244. They explicitly flag that the method works only for DiT architectures and that the overall correlation remains modest.

Critical strengths

The test setting makes sense. Poems and story prompts trip up reference-based metrics and CLIP-style similarity scores, and probing internal process signals is a legitimate research direction.

The authors do acknowledge the modest correlation and DiT-only limitation in their discussion. That restraint is welcome, yet it also reveals they already recognize their results are not strong enough to match the bolder tone set in the abstract.

Critical weaknesses

This metric is overclaimed and under-validated. The abstract and contributions frame it as a practical reference-free evaluator for creative text-to-image generation, but the supporting evidence is a weak Pearson correlation. It only narrowly outperforms standard baselines and sometimes falls behind them, and there is no experiment demonstrating downstream selection or practical utility. This is my core reason for recommending rejection.

There is also a conceptual mismatch, and the method feels incremental, packaged in a formulaic submission template. Their dominant-latent spatial statistics have no grounding in the input text, so they fail to address the stated concept-level research questions. Large sections come across as repetitive rephrasing: restating the same correlation numbers, the same limitations, and the same line about providing a complementary signal. Combined with caption and aggregation bugs, the manuscript reads more like an auto-generated draft than a polished ICLR submission.

Questions for authors

(These would need strong answers to change my rating; I am not optimistic they are quick fixes.)

  1. Why should the dominant SAE feature (by mean activation) correspond to prompt concepts at all? Please show text-grounded alternatives (e.g., features aligned to concept tokens) vs your argmax rule on the same human set.
  2. How do you score a single image in deployment? How many seeds are required, and how sensitive is r to m?
  3. Best-of-4 / pairwise accuracy vs human majority: does MIRAGE beat BLIP-2 in any decision task?
  4. Table 1 vs §4.9: equal weight or harmonic mean? Which number is in Table 2?
Reviewer_JvYSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Moving beyond concept-centered alignment evaluation, the paper proposes MIRAGE, a reference-free metric for creative text-to-image generation based on model-internal representations. MIRAGE first decomposes intermediate DiT activations with sparse autoencoder, and turns the dominant sparse feature into a single score that combines cross-seed spatial consistency and activation concentration through a harmonic mean. The metric is evaluated on poems and stories using SD3, FLUX, and PixArt.

Critical strengths

S1. The paper conducts human evaluations along four dimensions: semantic fidelity, attribute presence, compositional correctness, and narrative fidelity.

S2. The proposed reference-free metric captures two internal signals, cross-seed spatial consistency and activation concentration without requiring ground-truth images as references, which is particularly constructive for abstract generative tasks like poems and stories.

Critical weaknesses

W1. The research questions are vague.

  1. The paper does not explain how MIRAGE relates to the four human evaluation dimensions (semantic fidelity, attribute presence, compositional correctness, and narrative fidelity). It is unclear how the spatial statistics in Secs. 4.5–4.6 reflect any of them.

  2. The three RQs proposed in Sec. 1 are not revisited in Secs. 5–6, and the experiments and analyses do not lead to a clear conclusion for any of them.

W2. Writing and presentation issues.

  1. The flow from the problem formulation (Sec. 2) to the methodology (Sec. 4) is hard to follow. Sec. 2 introduces many equations without explanation, and Sec. 4 then redefines them in more detail, e.g., Eqs. (6)/(17)/(22), (7)/(24), and (10)/(28). The authors should either remove the repetition or clarify how the definitions differ.

  2. Secs. 5.1–5.4 all interpret Table 1, and the MIRAGE scores are restated in both Secs. 5.2 and 5.4. These subsections should be merged into one.

  3. Table 1 is duplicated as Table 3 (App. D), and Fig. 2 plots the same values.

  4. Figs. 1 and 3–6 appear to be AI-generated illustrations. This raises doubts about the faithfulness of the method.

W3. The results are of limited significance.

  1. In Table 1, the MIRAGE scores of FLUX, SD3, and PixArt (0.7866, 0.7959, and 0.7814) differ by less than 0.015, so the models are hard to distinguish given whatever the metric used in the study.
  2. The paper lacks in-depth analyses and ablations.
    • Table 4 does not demonstrate that harmonic aggregation is superior. All three aggregation methods rank SD3 highest, and their differences are marginal (≤ 0.025).
    • Although the testbeds are poems and stories (Sec. 3), all prompt-level cases (Figs. 4–6) are stories. Poems appear only by title in Fig. 3.
Questions for authors

Q1. How does MIRAGE relate to the four human evaluation dimensions?

Q2. Why do Sec. 5.5 and Table 2 report correlations only with the aggregated human score? The per-dimension results appear only in Table 10 (App. D), but this paper does not show the detailed results of comprehensive human evaluation on these testbeds.

Q3. Could the authors provide the input text for each case in Fig. 3? Each case is identified only by its title (e.g., "Good Bones"), which makes the alignment scores difficult to compare.

Q4. The caption of Fig. 4 states that the figure shows spatial centroids, component statistics, and the final MIRAGE score. Where are they? The figure contains only the prompt, the generated image, and heatmaps across denoising steps.

Reviewer_cdQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MIRAGE, a reference-free metric for evaluating images generated from creative texts such as poems and stories. MIRAGE uses the generator’s internal representations rather than directly comparing the final image with the text.

For each prompt, images are generated using four seeds, and image-token representations are extracted from selected Diffusion Transformer blocks and denoising steps. A separate sparse autoencoder with 4,096 features is trained for each selected block. For each seed, block, and step, the feature with the highest mean activation over image tokens is selected, and its activations are converted into a 64×64 spatial map.

MIRAGE combines two quantities: cross-seed centroid consistency, which measures the stability of activation locations across seeds, and activation concentration, which measures spatial localization within each map. These quantities are aggregated across blocks and steps, and their harmonic mean produces a prompt–generator-level score.

Critical strengths
  1. Evaluating images generated from poems and stories is challenging because these texts allow multiple valid visual interpretations and generally lack reference images. Investigating evaluation signals beyond direct output-level text–image comparisons is worthwhile.

  2. The paper explores whether spatial statistics of a generator’s internal features can provide a complementary evaluation signal. Experiments across three generator families and comparisons with human judgments provide an initial investigation of this direction.

Critical weaknesses

W1. It is unclear that MIRAGE measures text–image alignment. The authors select the feature with the highest mean activation rather than using an explicit text-token-to-feature correspondence (Sec. 4.4). The concept set C(P) defined in Sec. 2 does not appear to be used in the metric computation, and the feature is selected separately for each seed. Mean activation concentration stays at 0.66–0.71 across models and blocks (Tables 1, 5–7), but no sample-level distribution or reference value is reported. Cross-seed consistency may penalize legitimate compositional diversity, and its prompt-level definition leaves unclear how MIRAGE evaluates individual images. Basic sanity checks and quality metrics for the sparse autoencoders, such as reconstruction error and sparsity, are also missing. Without these, the correlation with human ratings cannot confidently be attributed to the proposed mechanism.

W2. The evidence for agreement with humans is limited, and several reported results require clarification. The overall correlation is r = 0.280, and on PixArt-α, MIRAGE (0.165) is below BLIP-2 (0.193). No confidence intervals or significance tests are reported. Tables 2 and 10 also require reconciliation: assuming the same samples and aggregation procedure, the reported aggregate and per-dimension correlations would require negative correlations among some human evaluation dimensions. In Figure 3, the same “SD3, Good Bones” image receives three different MIRAGE values (0.34/0.29/0.30). Although Appendix D.3 partially describes the human evaluation, important details remain unspecified, including the annotator count, rating scale, and which seed’s image was rated. Comparisons with learned preference metrics (ImageReward, PickScore, HPSv2) and per-component evaluations against human judgments are also missing. These gaps limit confidence in the claimed agreement with human judgments.

Questions for authors
  1. Please clarify the relationship between the aggregate correlations in Table 2 (0.280 overall; FLUX 0.351) and the per-dimension correlations in Table 10. Although correlation with an average can exceed individual correlations, the reported values would require negative correlations among some dimensions if computed on the same samples using the stated aggregation procedure. Please report the correlation matrix among the four dimensions and confirm that both tables use the same 579 prompt–model samples and corresponding model-specific subsets.

  2. Beyond the information provided in Appendix D.3, please specify the number of annotators, the rating scale, the annotation instructions, and the number of doubly annotated samples. Please also explain how the 193 prompts were selected, how long prompts are handled, and which denoising steps and exact FLUX blocks are used.

1 comment

Thank you for the detailed rebuttal and revised manuscript. The additional baselines and SAE diagnostics address several of the reviewer’s concerns. However, the revised manuscript still reports different per-dimension correlations from those stated in the rebuttal. Furthermore, reframing MIRAGE as a spatial diagnostic clarifies its scope but does not establish why its spatial statistics reliably indicate generation quality. These remaining concerns about the metric’s validity and the consistency of its evaluation lead the reviewer to maintain the original rating.

Comments on the paper

1 comment

I think reply of the author is not reliable and has a lot of extra efforts to improve their manuscript, so I will lower my score and tend to reject this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.