- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper argues that existing text-to-image metrics (CLIP, VQAScore, preference models) evaluate creator-side prompt alignment with the target text available, but do not measure whether an observer without the prompt can recover the creator's intended meaning. It proposes Prompt-Agnostic Intent Recovery (PAIR): a VLM observer (GPT-5.5) writes an open-ended description of the generated image without access to the prompt, and a separate LLM scorer (DeepSeek-V4-Pro) grades the description against a pre-specified, LLM-generated intent rubric (2–5 signals per intent). The authors build a benchmark of 300 intents across seven intent families and 1,800 images from two generators. They report that (i) recovery varies strongly across intent families (concrete control 9.27 vs. intentionality 3.47), (ii) PAIR-based selection among six candidates yields 9–17% higher held-out PAIR than CLIP, VQAScore, and a target-aware VLM selector, (iii) these findings transfer across observer/scorer families, (iv) human studies support the selections (85.9% vs. 39.4% intent identification on selector-disagreement cases), and (v) PAIR-guided refinement improves the final PAIR score from 7.12 to 8.77.
Critical strengths
- Extensive human evaluation. The paper complements the automatic protocol with five human studies
- Reproducibility. Code, intents, rubrics, recorded observer descriptions, score files, and analysis scripts are released
- Clear positioning against the cited related work. The related work section makes a deliberate effort to articulate how PAIR differs from each neighboring line of research it discusses
Critical weaknesses
W1. Unclear motivation. The examples mix distinct failure types. In "deliberate spill" and "reluctant handshake," the pose and expression do not convey the prompted intent, which is a generation/alignment failure rather than a new problem. Mooninite is a correctly realized artifact misread due to context, and "dawn vs. dusk" is inherent ambiguity. Sec. 10 itself admits the data cannot separate generator limitations from ambiguity. Please show cases that are fully prompt-aligned yet poorly recovered.
W2. Limited novelty and missing related work. The protocol amounts to "caption without the prompt, then compare with the target," which is used in many prior works with slightly different or sometimes similar goals:
- LLMScore (Lu et al., NeurIPS 2023): caption + LLM judge.
- DA-Score (Singh & Zheng, NeurIPS 2023): assertion decomposition + refinement.
- Idea2Img (Yang et al., ECCV 2024): VLM-feedback refinement.
- CycleReward (Bahng et al., ICCV 2025): image-to-text cycle consistency for best-of-N.
- PRIS/EFC (Kim et al., CVPR 2026): prompt decomposition + caption-based verification, best-of-N, prompt refinement.
W3. Circular evaluation and weak baselines. PAIR-selected images are evaluated with PAIR. "Held-out" is just another call to the same observer, scorer, and rubric, so the 9–17% gain mainly reflects test–retest consistency. Since intents are implicit, describe-then-score-per-signal naturally beats holistic CLIP/VQAScore. Fair baselines are needed: per-rubric-signal VQA (VQAScore/TIFA/DSG) and caption-based judges (LLMScore, PRIS/EFC).
W4. Sec. 4 does not identify why recovery is low. Without per-family alignment scores and per-family human recovery, low scores (e.g., intentionality 3.47) cannot be attributed to the generator, the image's ambiguity, or the VLM observer.
W5. Refinement (Sec. 8) is not new and is self-evaluated. Feedback-driven prompt refinement is established (Idea2Img, DA-Score, PRIS, Self-Refine). The gain drops from +1.65 to +0.69 under an independent stack, and only +0.79 over self-guided refinement. Comparisons with existing refinement methods and reports on image quality and alignment after refinement are missing.
Questions for authors
Please address every concern in the Weakness Section.
2 comments
Additional evaluation addressing W3
We re-evaluated the frozen Table 2 selections across all 300 six-image pools, replacing both the evaluation observer and scorer with Qwen3-VL-32B-Instruct and Qwen3.8-27B while keeping the rubrics unchanged.
| Baseline | PAIR − baseline | 95% CI |
|---|---|---|
| CLIP | +0.50 | [0.27, 0.74] |
| VQAScore | +0.39 | [0.16, 0.62] |
| EFC (core-first) | +0.23 | [0.04, 0.43] |
| Matched target-aware VLM | +0.15 | [−0.02, 0.31] |
Intervals use 10,000 paired bootstrap resamples over intents. The gains over CLIP, VQAScore, and EFC persist with evaluation models that did not participate in selection. They therefore extend beyond test–retest consistency within the original model pair. We added the protocol and results to Section 5 and Appendix D.10.
Thank you for the careful review and references. The revision adds an informed audit, fine-grained baselines, and an independent human selection comparison.
W1: Motivation. Your examples point to a real distinction, and our revision draws it explicitly with a new informed audit: these cases are not generation failures. 11 auditors, given the original intent, rated visual evidence for each rubric signal on the 56 images underlying Figure 5 (right). In Figure 8 (left), the key relational signal "a warm gesture coldly ignored" received a mean evidence rating of 0.80 on a 0–1 scale. With the prompt in hand, the required cues are present: the generation succeeded by instruction-following standards. Yet prompt-free viewers selected the intended interpretation in only 28% of responses on the same image: successful generation and failed viewing coexist.
Good instruction following does not imply the absence of ambiguity. Alignment evaluation asks "did the output reflect the input?"; we ask "what can a viewer without the prompt read out?". Across the 28 alignment-selected images, five combine high evidence with low identification; seven have low evidence, consistent with the cases you describe (Appendix E.5, Table 20). A metric must separate "not depicted" from "depicted but not recovered" — this is what PAIR measures.
W2: Novelty and related work. The original submission framed the problem mainly from visual communication and intent understanding, and did not adequately discuss the direct methodological precedents of describe-then-score and per-element verification. We have added the five works you pointed out to Section 2. LLMScore evaluates prompt alignment from image descriptions; DA-Score verifies element-wise via assertion-level VQA; CycleReward uses cycle consistency to learn a selection reward; Idea2Img and PRIS demonstrate feedback-driven refinement. EFC is closest to PAIR: both start from blind descriptions and check element by element. EFC can return to the same image for targeted follow-up on unconfirmed elements, whereas PAIR keeps the observer's response obtained without any target prompt, because what the observer does not spontaneously report is also part of the recovery failure. Through the intent-family benchmark and human studies, this paper asks whether audiences can still fail to identify the intended meaning even when the prompt is short and the image contains supporting cues (W1). On the 28 diagnostic pairs, viewers without the generation prompt identified the intended meaning 14.8 percentage points more often with PAIR-selected images than with EFC-selected images (W3).
W3: Evaluation and baselines. Held-out calls separate PAIR's selection from its scoring. We measured test–retest behavior directly: across three independent scoring calls on 1,800 images, the mean selection-minus-held-out difference is +0.009 (no systematic inflation between selection and evaluation), and retest noise at the 300-intent-mean scale is 0.085, an order of magnitude below the held-out gain over CLIP (1.10, 95% CI [0.86, 1.33]; Appendix D.4). Gains over CLIP also remain positive in both transfer directions with an open-weight observer (+0.62, +0.66; Appendix D.5). Section 7 evaluates selection with independent human judgments, consistent with the external evaluation of selected outputs in CycleReward and PRIS/EFC: identification was 85.9% (PAIR) vs. 39.4% (CLIP/VQAScore) in the original comparison (Appendix E.3, Table 18) and 86.8% vs. 72.0% (+14.8 pp) in the EFC comparison (13 participants, 364 judgments; Appendix E.4, Table 19).
Section 5 (Table 2) adds the two fine-grained controls you suggested: EFC and per-signal VQA using the same rubric score 6.91 and 7.08 on held-out PAIR, versus 7.50 for PAIR selection; a holistic-scoring ablation gives 7.21 (Appendix C).
W4: Attributing low recovery. CLIP alignment and PAIR recovery do not always move together across families (Section 4, Appendix D.8, Table 13). A same-image summary (28 images, four per family; Appendix E.1, Table 16) shows low recovery also in human answers: on intentionality, mean human and VLM description recovery are 4.13 and 4.15. The W1 audit distinguishes the two cases per image (Appendix E.5).
W5: Refinement. Feedback-driven refinement has precedents; Section 8 is scoped as an application probe of PAIR feedback rather than a head-to-head benchmark against existing refinement methods, using our pipeline and internal controls. PAIR feedback scores 0.79 above the self-guided pipeline sharing the planner and iteration schedule under the main stack (Table 22), and +0.69 over repeated generation under an independent open-weight stack (Appendix F.3). Paired analyses over 300 intents show no clear change in CLIP similarity (−0.18) or LAION aesthetic score (+0.02) (Appendix F.4).