- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DEPICT, a training-free metric for evaluating image-text alignment. Unlike existing decomposed evaluation methods that rely on fixed-YES references, DEPICT measures the agreement between image-based and caption-based answers to generated verification questions. The agreement scores are weighted by the confidence of the textual answers and further combined with a holistic alignment score.
The authors evaluate the method on five benchmarks using eleven VLM backbones from different model families. The results demonstrate competitive human-judgment correlations and substantial improvements in handling negated descriptions.
Critical strengths
-
The paper evaluates multiple training-free metrics across five benchmarks and eleven backbones, covering different model families, scales, and architectures. This systematic evaluation provides valuable evidence of the method's generalizability. The component ablations and confidence interval analyses further strengthen the experimental study.
-
The paper identifies a meaningful limitation of fixed-YES scoring and addresses it through answer agreement. The improvement on NegBench is substantial compared with fixed-YES baselines, and the results across different backbones support the effectiveness of the proposed scoring mechanism. The combination with holistic scoring also provides consistent benefits on several human-correlation benchmarks.
Critical weaknesses
-
The main contribution is to replace fixed-YES references with probabilistic agreement between textual and visual answers. However, earlier methods such as TIFA already use caption-derived reference answers rather than assuming YES for every question.
-
Although DEPICT achieves competitive results, its improvements over the strongest training-free baselines are relatively small on the human-correlation benchmarks. On Qwen3.5-27B, the SRCC gains are +0.022 on GenAI-Bench, +0.018 on TIFA160, and +0.011 on RichHF.
Questions for authors
Please see the critical weaknesses above. If the authors can convincingly address these concerns, I would be willing to reconsider and potentially raise my score.
Also, the authors evaluate Qwen3.5, Qwen3.6, and Qwen3.8 at the same model scale. Could the authors clarify the key differences among these closely related model versions and the motivation for selecting them? In particular, what additional insights does this comparison provide beyond the evaluations across different model families and scales?