acceptodds
Under review as a conference paper at ICLR 2027

From Prompt Alignment to Intent Recovery: Rethinking Text-to-Image Evaluation

Abstract

Text-to-image models have made substantial progress in faithfully rendering creators' prompts. However, ambiguous visual cues may lead observers to misinterpret the creator's intent from images alone, potentially triggering public alarm, reputational damage, and economic losses. Existing prompt-conditioned metrics primarily evaluate creator-side prompt alignment but do not directly measure observer-side intent recovery. Hence, we introduce **Prompt-Agnostic Intent Recovery** (PAIR), a protocol for evaluating intent recovery from generated images alone. A prompt-agnostic observer produces an open-ended image description and a separate scorer evaluates the description against a pre-specified intent rubric. We construct a benchmark of 300 intents and 1,800 images from two generators. Across the benchmark, prompt alignment and intent recovery induce different candidate rankings. When selecting among six images per intent from two generators, PAIR-based selection achieves 9-17% higher recovery on held-out evaluations than CLIP, VQAScore, and a target-aware VLM selector. In diagnostic selector-disagreement cases, participants without prompt access achieved 85.9% two-choice accuracy for PAIR-selected images versus 39.4% for alignment-selected images. Finally, PAIR-guided refinement improves the mean final PAIR score by 23% over repeated uncalibrated generation. These results identify observer-side intent recovery as an evaluation target not captured by prompt-conditioned alignment.

Primary area:
datasets and benchmarks
Keywords:
text-to-image generation, multimodal evaluation, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_12vJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that existing text-to-image metrics (CLIP, VQAScore, preference models) evaluate creator-side prompt alignment with the target text available, but do not measure whether an observer without the prompt can recover the creator's intended meaning. It proposes Prompt-Agnostic Intent Recovery (PAIR): a VLM observer (GPT-5.5) writes an open-ended description of the generated image without access to the prompt, and a separate LLM scorer (DeepSeek-V4-Pro) grades the description against a pre-specified, LLM-generated intent rubric (2–5 signals per intent). The authors build a benchmark of 300 intents across seven intent families and 1,800 images from two generators. They report that (i) recovery varies strongly across intent families (concrete control 9.27 vs. intentionality 3.47), (ii) PAIR-based selection among six candidates yields 9–17% higher held-out PAIR than CLIP, VQAScore, and a target-aware VLM selector, (iii) these findings transfer across observer/scorer families, (iv) human studies support the selections (85.9% vs. 39.4% intent identification on selector-disagreement cases), and (v) PAIR-guided refinement improves the final PAIR score from 7.12 to 8.77.

Critical strengths
  • Extensive human evaluation. The paper complements the automatic protocol with five human studies
  • Reproducibility. Code, intents, rubrics, recorded observer descriptions, score files, and analysis scripts are released
  • Clear positioning against the cited related work. The related work section makes a deliberate effort to articulate how PAIR differs from each neighboring line of research it discusses
Critical weaknesses

W1. Unclear motivation. The examples mix distinct failure types. In "deliberate spill" and "reluctant handshake," the pose and expression do not convey the prompted intent, which is a generation/alignment failure rather than a new problem. Mooninite is a correctly realized artifact misread due to context, and "dawn vs. dusk" is inherent ambiguity. Sec. 10 itself admits the data cannot separate generator limitations from ambiguity. Please show cases that are fully prompt-aligned yet poorly recovered.

W2. Limited novelty and missing related work. The protocol amounts to "caption without the prompt, then compare with the target," which is used in many prior works with slightly different or sometimes similar goals:

  • LLMScore (Lu et al., NeurIPS 2023): caption + LLM judge.
  • DA-Score (Singh & Zheng, NeurIPS 2023): assertion decomposition + refinement.
  • Idea2Img (Yang et al., ECCV 2024): VLM-feedback refinement.
  • CycleReward (Bahng et al., ICCV 2025): image-to-text cycle consistency for best-of-N.
  • PRIS/EFC (Kim et al., CVPR 2026): prompt decomposition + caption-based verification, best-of-N, prompt refinement.

W3. Circular evaluation and weak baselines. PAIR-selected images are evaluated with PAIR. "Held-out" is just another call to the same observer, scorer, and rubric, so the 9–17% gain mainly reflects test–retest consistency. Since intents are implicit, describe-then-score-per-signal naturally beats holistic CLIP/VQAScore. Fair baselines are needed: per-rubric-signal VQA (VQAScore/TIFA/DSG) and caption-based judges (LLMScore, PRIS/EFC).

W4. Sec. 4 does not identify why recovery is low. Without per-family alignment scores and per-family human recovery, low scores (e.g., intentionality 3.47) cannot be attributed to the generator, the image's ambiguity, or the VLM observer.

W5. Refinement (Sec. 8) is not new and is self-evaluated. Feedback-driven prompt refinement is established (Idea2Img, DA-Score, PRIS, Self-Refine). The gain drops from +1.65 to +0.69 under an independent stack, and only +0.79 over self-guided refinement. Comparisons with existing refinement methods and reports on image quality and alignment after refinement are missing.

Questions for authors

Please address every concern in the Weakness Section.

2 comments

Thank you for the careful review and references. The revision adds an informed audit, fine-grained baselines, and an independent human selection comparison.

W1: Motivation. Your examples point to a real distinction, and our revision draws it explicitly with a new informed audit: these cases are not generation failures. 11 auditors, given the original intent, rated visual evidence for each rubric signal on the 56 images underlying Figure 5 (right). In Figure 8 (left), the key relational signal "a warm gesture coldly ignored" received a mean evidence rating of 0.80 on a 0–1 scale. With the prompt in hand, the required cues are present: the generation succeeded by instruction-following standards. Yet prompt-free viewers selected the intended interpretation in only 28% of responses on the same image: successful generation and failed viewing coexist.

Good instruction following does not imply the absence of ambiguity. Alignment evaluation asks "did the output reflect the input?"; we ask "what can a viewer without the prompt read out?". Across the 28 alignment-selected images, five combine high evidence with low identification; seven have low evidence, consistent with the cases you describe (Appendix E.5, Table 20). A metric must separate "not depicted" from "depicted but not recovered" — this is what PAIR measures.

W2: Novelty and related work. The original submission framed the problem mainly from visual communication and intent understanding, and did not adequately discuss the direct methodological precedents of describe-then-score and per-element verification. We have added the five works you pointed out to Section 2. LLMScore evaluates prompt alignment from image descriptions; DA-Score verifies element-wise via assertion-level VQA; CycleReward uses cycle consistency to learn a selection reward; Idea2Img and PRIS demonstrate feedback-driven refinement. EFC is closest to PAIR: both start from blind descriptions and check element by element. EFC can return to the same image for targeted follow-up on unconfirmed elements, whereas PAIR keeps the observer's response obtained without any target prompt, because what the observer does not spontaneously report is also part of the recovery failure. Through the intent-family benchmark and human studies, this paper asks whether audiences can still fail to identify the intended meaning even when the prompt is short and the image contains supporting cues (W1). On the 28 diagnostic pairs, viewers without the generation prompt identified the intended meaning 14.8 percentage points more often with PAIR-selected images than with EFC-selected images (W3).

W3: Evaluation and baselines. Held-out calls separate PAIR's selection from its scoring. We measured test–retest behavior directly: across three independent scoring calls on 1,800 images, the mean selection-minus-held-out difference is +0.009 (no systematic inflation between selection and evaluation), and retest noise at the 300-intent-mean scale is 0.085, an order of magnitude below the held-out gain over CLIP (1.10, 95% CI [0.86, 1.33]; Appendix D.4). Gains over CLIP also remain positive in both transfer directions with an open-weight observer (+0.62, +0.66; Appendix D.5). Section 7 evaluates selection with independent human judgments, consistent with the external evaluation of selected outputs in CycleReward and PRIS/EFC: identification was 85.9% (PAIR) vs. 39.4% (CLIP/VQAScore) in the original comparison (Appendix E.3, Table 18) and 86.8% vs. 72.0% (+14.8 pp) in the EFC comparison (13 participants, 364 judgments; Appendix E.4, Table 19).

Section 5 (Table 2) adds the two fine-grained controls you suggested: EFC and per-signal VQA using the same rubric score 6.91 and 7.08 on held-out PAIR, versus 7.50 for PAIR selection; a holistic-scoring ablation gives 7.21 (Appendix C).

W4: Attributing low recovery. CLIP alignment and PAIR recovery do not always move together across families (Section 4, Appendix D.8, Table 13). A same-image summary (28 images, four per family; Appendix E.1, Table 16) shows low recovery also in human answers: on intentionality, mean human and VLM description recovery are 4.13 and 4.15. The W1 audit distinguishes the two cases per image (Appendix E.5).

W5: Refinement. Feedback-driven refinement has precedents; Section 8 is scoped as an application probe of PAIR feedback rather than a head-to-head benchmark against existing refinement methods, using our pipeline and internal controls. PAIR feedback scores 0.79 above the self-guided pipeline sharing the planner and iteration schedule under the main stack (Table 22), and +0.69 over repeated generation under an independent open-weight stack (Appendix F.3). Paired analyses over 300 intents show no clear change in CLIP similarity (−0.18) or LAION aesthetic score (+0.02) (Appendix F.4).

AuthorsOpenReview ↗

Additional evaluation addressing W3

We re-evaluated the frozen Table 2 selections across all 300 six-image pools, replacing both the evaluation observer and scorer with Qwen3-VL-32B-Instruct and Qwen3.8-27B while keeping the rubrics unchanged.

BaselinePAIR − baseline95% CI
CLIP+0.50[0.27, 0.74]
VQAScore+0.39[0.16, 0.62]
EFC (core-first)+0.23[0.04, 0.43]
Matched target-aware VLM+0.15[−0.02, 0.31]

Intervals use 10,000 paired bootstrap resamples over intents. The gains over CLIP, VQAScore, and EFC persist with evaluation models that did not participate in selection. They therefore extend beyond test–retest consistency within the original model pair. We added the protocol and results to Section 5 and Appendix D.10.

Reviewer_q33NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that traditional evaluation metrics for T2I generation primarily focus on the alignment between the image and the prompt, but overlook whether a human can understand the creator's true intent only based on the image. To address this, the authors propose the PAIR, which assesses an image's ability to convey intent by distinguishing between observers who are unaware of the promp and rater who are informed of the intended goal. The authors construct a benchmark comprising 300 intents and 1,800 images, and have validated the effectiveness of PAIR through comparisons, human studies, and iterative optimization studies.

Critical strengths
  1. This paper addresses a significant and under-researched problem in the evaluation of T2I generation that prompt consistency is not necessarily equal to the creator's original intent. By redefining the evaluation objective from the perspective of an observer, PAIR offers both novelty and practical utility.
  2. The design of PAIR strictly separates the information accessible to the observer from that available to the evaluator to minimize the bias that prior knowledge of the prompt might introduce into the interpretation of an image. The authors develop a benchmark covering 7 distinct categories of intent to provide a systematic foundation for understanding this problem.
  3. The experiment is compehensive. The authors assess the framework's stability across various observer and evaluator models and validate key findings through human studies.
Critical weaknesses
  1. The paper primarily compares its approach against earlier evaluation methods, such as CLIP and VQAScore.
  2. In the image selection experiments, PAIR is used both as the selection metric and as the final evaluation criterion. Although the authors employ a held-out evaluation strategy to mitigate assessment noise, this experimental setup inherently favors PAIR.
  3. The computational cost of PAIR is potentially high. Compared to lightweight metrics like CLIP, PAIR uses a vision-language model to generate descriptions and another vision-language model for scoring. Moreover, in some experiments, PAIR involves multiple rounds of evaluation. The paper lacks a clear analysis of the trade-offs between computational cost, inference latency, and performance gains.
Questions for authors

Refer to Critical Weaknesses.

3 comments

Thank you for the review. Following your comments, the revision adds fine-grained baselines, an independent human comparison against EFC, and measured per-pool scoring time.

W1: Earlier baselines. We added a recent fine-grained verifier, EFC (Kim et al., CVPR 2026), to Section 5 (Table 2), together with a direct per-signal VQA using the same intent rubric as PAIR. On the same 300 six-candidate pools, mean held-out PAIR scores are 6.91 (EFC), 7.08 (per-signal VQA), and 7.50 (PAIR). EFC and PAIR pick the same image in 47.0% of pools. Our EFC implementation reuses PAIR's first-pass blind descriptions and ranks candidates by the EFC paper's core-first rule. Full implementation details are in Appendix C.

W2: Selection and independent evaluation. The held-out design separates the PAIR calls used for selection from those used for scoring, and the automated results still measure performance under the same scoring protocol (Section 5, Table 2; Appendix D.4, Table 6). Section 7 additionally compares selection with independent human judgments, following the practice of validating against human judgments in CycleReward and EFC, and extending it from validating the scoring metric to validating the selection outcome itself. On the existing 28 CLIP/VQAScore disagreement pairs, prompt-free identification was 85.9% for PAIR-selected vs. 39.4% for baseline-selected images (Appendix E.3, Table 18). On the new 28 EFC–PAIR disagreement pairs, 13 participants made 364 two-alternative judgments, with identification of 86.8% vs. 72.0%, a difference of 14.8 percentage points (95% CI [+1.4, +28.3]; Appendix E.4, Table 19). Each result applies to its own diagnostic sample selected by held-out PAIR gaps. The mean selection-minus-held-out difference over 1,800 images is +0.009, with retest noise of 0.085 at the 300-intent-mean scale, an order of magnitude below the gain over CLIP (Appendix D.4).

W3: Computational cost. Appendix C adds call counts and measured scoring time. A single PAIR scoring of one image costs one visual-observation call and one text-scoring call. With GPT-5.5 as observer and DeepSeek as scorer, and image-level parallelism across the six candidates, one scoring pass per pool requires 12 calls with a median measured time of 77 s (six measurements, range 47–136 s). The two-pass averaged selection setting used in this paper requires 24 calls, with a median scoring time of about 172 s over three pools (range 100–232 s, excluding the final ranking step and the held-out evaluation). Iterative refinement additionally involves image-generation and planner calls (Appendix F). The human results in Section 7 show that, on the tested disagreement samples, this extra evaluation step provides audience-side evidence for pre-publication selection.

AuthorsOpenReview ↗

Additional evaluation addressing W2

To further address the selection–evaluation concern, we re-evaluated the frozen Table 2 selections across all 300 six-image pools using Qwen3-VL-32B-Instruct as observer and Qwen3.8-27B as scorer, replacing both evaluation models while keeping the rubrics unchanged.

BaselinePAIR − baseline95% CI
CLIP+0.50[0.27, 0.74]
VQAScore+0.39[0.16, 0.62]
EFC (core-first)+0.23[0.04, 0.43]
Matched target-aware VLM+0.15[−0.02, 0.31]

Intervals use 10,000 paired bootstrap resamples over intents. PAIR retains its advantages over CLIP, VQAScore, and EFC even when evaluated by a different observer and scorer, neither of which participated in selection. We added the protocol and results to Section 5 and Appendix D.10.

Thank you for the detailed response. I will maintain my positive score.

Reviewer_4peGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of how observers recover intent from generated images, and proposes an evaluation protocol named Prompt-Agnostic Intent Recovery (PAIR) to evaluate such intent discovery from the generated images. The contributed benchmark consists of 300 intents and 1800 images from two image generators, annotated with the scores of the observer description. The experiments show that prompt alignment and intent discovery can yield different rankings of candidate images, underlining the potential of intent discovery as a new metric.

Critical strengths

S1. This paper proposes a new perspective on evaluating the generated images from observers without access to the prompt. By formulating the process in terms of the prompt, the generated image, and the intent recovered from the observer, this paper presents an interesting metric and aspect of text-to-image evaluation.

S2. This paper also contributes a benchmark and demonstrates that intent recovery differs from prompt alignment metric to show the potential of evaluating the intent recovery for the generated images. The human study also collected extensive results which show a high correlation with VLM results.

Critical weaknesses

W1. Although the intent recovery sounds like a novel concept, it remains unclear how this work distinguishes or disentangles the caption or pure description from the user intent. The contribution would be weakened if the observers describe too much of the image content rather than the hidden intent.

W2. The downstream application of the new metric is not well motivated. This work shows that intent discovery differs from prompt alignment and its use as a refinement signal, which is good, but might not sufficient to demonstrate why it's necessary to use the new metric. More empirical evidence on practical scenarios or downstream tasks could strengthen this claim.

Minor: The presentation or the flow of this work could be further strengthened. Some sections and paragraphs took a little more time to parse due to a mixture of discussion and a lot of results and confidence intervals presented in numbers. It might be better to present these numerical results in a table or other format instead of extensively listing them in paragraphs.

Questions for authors

The major concerns that affect my assessments are listed under critical weaknesses and here are clarification questions. Q1. What if the prompt for text-to-image generation includes the intent from the creator? Will it effectively improve intent recovery?

3 comments

Thank you for the helpful review. We have further clarified the distinction between scene description and intent recovery, strengthened the downstream motivation with new human selection evidence, and reorganized the results sections in the revision.

W1: Description vs. intent recovery. In PAIR, the observer sees only the image, and the scorer only checks whether the response supports the pre-specified intent signals. Extra visual detail counts only when it supports these signals. For example, the intent of the image in Figure 8 (left) is "a person reaching out warmly to another, only to be coldly ignored." The blind observer read the gesture as explaining, asking, or persuading, and described the other person as listening; the warm-gesture signal scored 0 in two of three scoring calls, and the answer never expressed the "coldly ignored" relation. Scoring the same descriptions with the correct rubric averages 7.19, versus 0.73 under a same-family rubric of another intent (Appendix D.3). In addition, human and VLM per-intent mean recovery scores correlate positively over the 28 intents of the open-ended human study (Spearman ρ = 0.715; Section 7, Appendix E.1).

W2: Downstream motivation. Once an image is shown without its generation prompt, a creator wants audiences to read the intended meaning, and to catch likely misreadings in advance. The Boston Mooninite incident in Figure 1 is this problem in its full form. In 2007, the devices ran in multiple cities for weeks, and the cartoon character on each device fully realized the creative intent. Any text–image alignment check, any element-by-element self-check by the creators, would pass. In Boston, passers-by without the prompt read the same devices as suspected explosives: bomb squads deployed, transit shut down, and the case settled for $2M. Note the structure: every creator-side check passed; the failure happened exactly at the prompt-absent interpretation step, the one step that traditional metrics structurally cannot cover and that our metric directly measures. This indicates that prompt-free recovery screening can surface such high-confidence competing readings before release. The incident predates T2I, which shows the risk is real and carries monetary costs; generative pipelines are now industrializing it: intent-laden images are mass-produced while the creator's only self-check is prompt alignment. Similarly, a public-signage designer can collect first-time visitors' readings of a stylized wayfinding sign to see whether it sends people in the wrong direction, and decide whether to fix the graphic or add text. Our metric provides the measurement for exactly this kind of pre-publication screening.

The new experiment tests the selection problem behind these scenarios. On 28 EFC–PAIR diagnostic pairs (selected by intent family and held-out PAIR gap), prompt-free participants identified the intended meaning in 86.8% of cases for PAIR-selected images versus 72.0% for EFC-selected images, a gap of 14.8 percentage points. On the tested sample, audience-side intent identification still provides additional evidence for selection even on top of fine-grained prompt verification (Section 7, Appendix E.4, Table 19).

Q: Writing the intent into the generation prompt. Our benchmark already instantiates this condition: the full verbalized intent is written into every generation prompt, and the observer sees only the generated image. Even under this condition, mean PAIR is 9.27 for concrete control but 3.47 for intentionality (Section 4). So writing the intent into the prompt can improve alignment, but it does not guarantee recovery: even with the intent explicitly written in, audiences may still fail to recover it from the image.

Minor: presentation. We consolidated the results: selection results are in Table 2 (Section 5); cross-model comparisons, human selection studies, and refinement statistics are in Figure 6 and Appendices D.5–D.6, Appendices E.3–E.4, and Appendix F, respectively. The main text is organized around the question each experiment answers, with detailed statistics consolidated in tables and the appendix.

Thank you for the detailed responses. The authors’ responses to W1 and Q1 have resolved my concerns. However, the authors addressed W2 and the refinement experiments mainly by narrowing their claims, and, as also noted by Reviewer 12vJ, the method remains only marginally different from existing describe-then-score approaches (e.g., EFC). I therefore maintain my original score. For the revision, I would suggest supporting the downstream motivation with evidence from a concrete application setting.

Thank you for confirming that W1 and Q1 are resolved. Regarding EFC, the distinction is the evaluation endpoint: EFC verifies target elements and can use target-informed follow-up, whereas PAIR measures the interpretation obtained without target guidance. Our benchmark and human studies examine the practical consequences of this distinction.

The downstream decision we test is selecting images that viewers can interpret without the generation prompt. On the 28 EFC–PAIR diagnostic pairs, human identification is 86.8% for PAIR selections versus 72.0% for EFC (+14.8 percentage points, 95% CI [1.4, 28.3]; Appendix E.4).

In our 18 November follow-up to Reviewer 12vJ, we also added a direct, generation-budget-matched comparison with adapted Idea2Img on 28 stratified intents (Section 8, Appendix F.5). Under the independent Qwen stack, PAIR-guided refinement scores 8.23 versus 7.51 (paired difference +0.71, 95% CI [−0.22, 1.67]). This complements the original 300-intent refinement study by adding an established-method reference for the use of PAIR feedback.

A concrete application setting we explored early in this work is visual word-puzzle design: turning an abstract phrase into an image that players can decode. For a target such as “no idea,” PAIR-guided refinement can explore visual expressions such as “no-eye deer,” using target-blind interpretations as feedback. The application tests whether this process produces images from which players can recover the intended phrase. Following your suggestion, if the paper is accepted, we will incorporate this application case study into the camera-ready version, with task-specific evaluation of how PAIR supports visual communication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.