acceptodds
Withdrawn from ICLR 2027

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-Text Alignment, training-free VLM metrics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P6VvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEPICT, a training-free metric for evaluating image-text alignment. Unlike existing decomposed evaluation methods that rely on fixed-YES references, DEPICT measures the agreement between image-based and caption-based answers to generated verification questions. The agreement scores are weighted by the confidence of the textual answers and further combined with a holistic alignment score.

The authors evaluate the method on five benchmarks using eleven VLM backbones from different model families. The results demonstrate competitive human-judgment correlations and substantial improvements in handling negated descriptions.

Critical strengths
  1. The paper evaluates multiple training-free metrics across five benchmarks and eleven backbones, covering different model families, scales, and architectures. This systematic evaluation provides valuable evidence of the method's generalizability. The component ablations and confidence interval analyses further strengthen the experimental study.

  2. The paper identifies a meaningful limitation of fixed-YES scoring and addresses it through answer agreement. The improvement on NegBench is substantial compared with fixed-YES baselines, and the results across different backbones support the effectiveness of the proposed scoring mechanism. The combination with holistic scoring also provides consistent benefits on several human-correlation benchmarks.

Critical weaknesses
  1. The main contribution is to replace fixed-YES references with probabilistic agreement between textual and visual answers. However, earlier methods such as TIFA already use caption-derived reference answers rather than assuming YES for every question.

  2. Although DEPICT achieves competitive results, its improvements over the strongest training-free baselines are relatively small on the human-correlation benchmarks. On Qwen3.5-27B, the SRCC gains are +0.022 on GenAI-Bench, +0.018 on TIFA160, and +0.011 on RichHF.

Questions for authors

Please see the critical weaknesses above. If the authors can convincingly address these concerns, I would be willing to reconsider and potentially raise my score.

Also, the authors evaluate Qwen3.5, Qwen3.6, and Qwen3.8 at the same model scale. Could the authors clarify the key differences among these closely related model versions and the motivation for selecting them? In particular, what additional insights does this comparison provide beyond the evaluations across different model families and scales?

Reviewer_ZsFFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEPICT, a training-free metric for evaluating text-to-image alignment that addresses the fixed-YES assumption in existing decomposed evaluation methods. It measures the agreement between image-based and caption-only answers to fine-grained verification questions, with a commitment-based weighting mechanism to account for question reliability. The resulting decomposed score is further combined with holistic evaluation to capture both fine-grained details and global semantic consistency.

Critical strengths

1.The paper is clearly written, with well-organized figures and tables that effectively illustrate the proposed method and experimental results.

2.It identifies a meaningful limitation of existing decomposed T2I alignment metrics and introduces a simple yet effective agreement-based scoring strategy to address it. The proposed DEPICT framework is training-free and can be readily applied to different VLM backbones. Moreover, combining fine-grained verification with holistic evaluation effectively leverages their complementary strengths. The experiments are comprehensive, covering five benchmarks and eleven backbones, with extensive ablation studies and statistical analyses.

Critical weaknesses
  1. The core methodological contribution of this paper appears to be relatively limited, as the proposed approach primarily introduces an answer agreement mechanism to combine image-based and caption-only responses for decomposed verification questions. It is somewhat unclear whether the technical novelty and scope of the proposed method are sufficient to meet the expectations of an ICLR paper. Furthermore, text-to-image alignment may no longer represent one of the most pressing challenges for current state-of-the-art T2I generation models. Therefore, the practical significance of studying this evaluation dimension as a standalone problem could be further justified. It might be more appropriate to incorporate this dimension into a broader and more comprehensive evaluation framework.

  2. The paper lacks systematic evaluations on long T2I generation prompts. Given that the proposed method relies on decomposing captions into individual verification questions, its effectiveness in handling lengthy descriptions with numerous semantic constraints remains insufficiently explored. It would be valuable to include experimental results on dedicated long-prompt T2I benchmarks, such as LPG-Bench (introduced in TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency) and LongT2IBench (LongT2IBench: A Benchmark for Evaluating Long Text-to-Image Generation with Graph-structured Annotations), to further demonstrate the robustness and generalizability of the proposed method.

  3. The authors are encouraged to provide a more detailed discussion and comparison with TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency. Although this work has not yet been formally published, it remains relevant to the research problem addressed in this paper. It would be helpful to clarify the methodological differences between TIT-Score and DEPICT and, if feasible, provide quantitative comparisons on common benchmarks to better demonstrate the relative advantages of the proposed approach.

Questions for authors

See weaknesses

Reviewer_n7axOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes DEPICT, a training-free metric for evaluating text-to-image alignment. The method breaks captions into questions, compares answers obtained from the image and caption, weights questions based on how confidently the caption determines their answers, and combines the resulting score with VQAScore.

Critical strengths

The paper is well written, and the experiments are extensive, covering five benchmarks and eleven VLM backbones. The method shows consistent improvements over existing training-free approaches.

Critical weaknesses

I find the novelty of the proposed approach relatively limited. As I understand it, DEPICT largely combines ideas from existing methods.

Breaking captions into questions was already introduced by methods such as TIFA/DSG, while the original TIFA also determined the correct answers from the caption rather than assuming that the answer is always “yes”. Soft-TIFA uses the probabilities of the image-based answers, and VQAScore already evaluates how well an image matches its entire caption.

The main differences therefore seem to be:

  1. Probabilistic agreement: Instead of selecting a single correct answer from the caption, DEPICT uses the probabilities of both YES and NO to calculate agreement between the caption-based and image-based answers.
  2. Confidence weighting: Questions are weighted based on how confidently the caption determines their answers. However, Table 3 suggests that this contributes very little to the final performance.
  3. Combination with VQAScore: The score obtained from the individual questions is averaged with VQAScore using a simple weighted average.

While the paper includes ablations of these components, an important comparison is missing. How does DEPICT compare against a method that determines a single correct YES/NO answer from the caption, as in the original TIFA, but still uses the image VLM’s probability of that answer? This baseline should also be combined with VQAScore to determine whether using the full probability distribution actually provides an advantage.

Without this comparison, it is difficult to determine whether the improvements come from the proposed approach or simply from correctly handling the expected answers, including negations. Furthermore, the improvements over the strongest existing baselines are relatively small on most benchmarks, particularly considering the additional computational cost.

Questions for authors

While the experiments are thorough, I am not convinced that the paper offers a sufficiently substantial contribution over existing methods. I would particularly like to see the comparison described above to better understand the advantage of the proposed scoring approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.