acceptodds
Under review as a conference paper at ICLR 2027

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

Abstract

Current image–text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explainable metrics provide rationales, but the visual evidence supporting those rationales can still be difficult to verify. We introduce VisExMEM, a reference-free evaluator that decomposes captions into atomic claims, localizes visual evidence, verifies each claim, and averages claim-level support into a global score. Explanations use the same evidence and decisions. We train Qwen3.5-based 4B and 9B models on 100K samples with multi-task SFT, counterfactual refinement, and evidence-aware GDPO. Across eight benchmarks, VisExMEM-9B achieves the highest descriptive composite among open-source evaluators in our comparison, exceeding VQAScore by 11.7 points while remaining competitive with GPT-5.6 Sol. We also introduce AGITA-Bench, containing 600 human-annotated image–text pairs and 8.7K atomic claims for global and claim-level evaluation. For untuned Qwen3.5 and GPT-5.6 Sol controls, decomposed prompting raises the same composite by 3.6–5.1 points relative to holistic prompting. Human evaluation shows that VisExMEM explanations outperform prior explainable metrics and approach GPT-5.6 Sol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal evaluation, image-text alignment, explainable evaluation, visual grounding, vision-language models, reference-free evaluation, compositional reasoning, text-to-image evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_L5zGOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors look at the task of text/image alignment. They break the text into entities and claims, and measure the existence/alignment of each in a the given image. They then provide an aggregate score over all claim evaluations. The authors train a model to perform claim evaluation based on the image, the claim and a bounding box. The authors evaluate their approach on several benchmarks, including a new one introduced in the paper, AGITA-BENCH.

  • Note: I commented on several baselines, ablations and analyses that I think are important for the paper to pass the quality bar. If the authors provide them and they still indicate at the novelty and quality of VisExMEM, I would raise my score.

  • Update: the authors have fully answered all my questions and comments. With the additional experiments and info I think the paper is well rounded and I am raising my score accordingly.

Critical strengths
  • The problem of image/text alignment eval is important to keep improving image generation and caption generation with increasingly complex inputs.

  • The proposed approach is novel and interesting

  • The training process is interesting and well written at the high level, though some fine details are missing, around training data generation for example, that are needed to reproduce the approach.

  • The evaluation against existing benchmarks and prior eval works is rigorous

Critical weaknesses
  • I am missing VNLI-like (see Yarom, 2023) baseline, a model trained on similar data as used for VisExMEM, but only to predict VNLI(image, caption) -> yes/no, without breaking into claims and aggregating.

  • VQ^2 also splits the text into claims and match each claim, but in Yarom, 2023, they did not train a model. Training VisExMEM without explanations and bounding boxes would be exactly like that. This would be a useful baseline/set of ablations to understand the impact of each of these inputs/output on the alignment decision quality.

  • I am missing error analyses on VisExMEM outputs, both the alignment decision at each claim type and overall, as well as the explanation quality.

  • I am missing an analysis of the quality of VisExMEM w.r.t caption complexity, e.g. the number of generated claims, their types etc. More specifically, is there a range of captions on which VisExMEM improves the most compared to baselines, or is it improving both on easy and difficult prompts alike.

  • AGITA-BENCH looks like an interesting benchmark, but it is impossible to evaluate its usefulness and quality based on half a page. Given that it is only one of many benchmarks evaluated in the paper, I would consider a separate paper about it, to measure its novelty, quality etc. I prefer more focus and space for ablations and analyses of VisExMEM than adding a benchmark whose quality is hard to evaluate from the paper. Overall, I see AGITA-BENCH as a weakness of the paper, not its strength.

Questions for authors
  • Please see my comments above

  • [1] also proposes to generate explanations, both textual and visual (bounding box) instead of getting the bounding box candidate as input. Could you please explain why not use this fully generative approach (part of y's tokens are the bounding box coordinates) instead of getting a bounding box as input? (a disclaimer - perhaps I didn't understand the paper and that's what you are doing)

[1] Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment

Reviewer_VJ8nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new model-based metric for evaluating the alignment between image and text. Specifically, it fine-tunes a base VLM to generate compositional claims and verify them based on visual evidence. A new human-annotated benchmark is also presented to facilitate research in the related field. Experimental results across multiple datasets show that the proposed metric achieves better assessment of the alignment over most open-source models.

Critical strengths

(1) The paper tackles an important problem on evaluating vision-language alignment. The proposed metric takes into account compositionality and visual grounding, providing a relatively interpretable way to quantify the model performance.

(2) A new benchmark is provided. The dataset emphasizes compositional rating and considers samples with diverse difficulties.

(3) The paper provides extensive analysis under different settings, which eases understanding the proposed metrics and prior methods from different perspectives.

(4) Experimental results show relatively consistent improvement of the proposed method, highlighting its generalizability over samples with distinct characteristics.

Critical weaknesses

(1) The paper proposes a multi-stage fine-tuning method, nevertheless, its contribution is not sufficiently validated. For instance, the performance gain can also be attributed to additional training (without the proposed pipeline). It would be important to consider additional baseline, e.g., QWEN3.5 trained with the same samples but with only holistic scores.

(2) Looking at Table 1 and Table 2, it appears that the proposed method has significantly lower performance on the new AGITA dataset. This is relatively counterintuitive to me, as the dataset is supposed to have a stronger emphasis on compositional evaluation.

(3) Stage 2 of the proposed method involves a counterfactual loss, however, there is no corresponding evaluation in the experiments. Including additional analysis regarding the sensitivity to counterfactual perturbation could strengthen the results.

(4) The proposed metric treats the scores for individual claims equally, which can be problematic. For instance, for an image with a person interacting with multiple objects and associating with different attributes, an incorrect prediction of the person’s characteristics (e.g., gender, age, etc) could potentially contaminate the scores for all other correlated objects/attributes despite their correctness.

Questions for authors

(1) Please consider disentangling the contributions of the proposed training algorithm and generic training with additional data.

(2) Why does the proposed method achieve the smallest improvement on the proposed compositionality-centric benchmark?

(3) Does the proposed method also enhance the faithfulness of evaluation (e.g., against counterfactual samples)?

(4) How does the method handle the potential issue about error aggregation?

Reviewer_Q5euOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to make image–text evaluators’ explanations more faithfully reflect the visual evidence and claim decisions. To this end, it introduces VisExMEM, which decomposes text into atomic claims, verifies each claim using localized evidence, and averages the support scores into a global score. Explanations are then generated from the same evidence and fixed claim decisions. The evaluator is trained through multi-task SFT, counterfactual refinement, and evidence-aware GDPO. The paper also introduces AGITA-Bench to assess agreement with human judgments at both global and atomic levels. Experiments demonstrate competitive scoring performance.

Critical strengths
  1. The paper presents a comprehensive evaluation pipeline connecting atomic claim decomposition, visual evidence selection, claim-level verification, score aggregation, and explanation generation. This design makes individual scoring decisions easier to inspect.
  2. AGITA-Bench is a valuable addition, providing human annotations at both global and atomic levels. This enables separate assessment of overall scoring and claim-level semantic verification, helping diagnose evaluator performance beyond aggregate scores.
Critical weaknesses
  1. The paper’s readability could be improved. Several important quantities and objectives—including the Stage 2 loss, the reward (r_{i,g}) in Equation (3), and (L_{\mathrm{causal}}) in the Stage 3 objective at line 241—are defined only in the appendix. These definitions are necessary in the main text to make the method self-contained and easier to follow. Additionally, the Global H/V value for the Easy·I2T example in Figure 4 appears to be missing a decimal point and should read 0.75/0.79.
  2. I have concerns about the paper’s central motivation. At line 45, the paper identifies a remaining issue as “whether the explanation reflects the visual evidence and the decisions that produced the score.” Accordingly, I understand the paper as aiming to provide more faithful explanations by generating them from visual evidence and decisions, without allowing the explanations to alter the score. However, the experiments primarily focus on scoring performance. Although a human evaluation assesses explanation quality, this is insufficient to establish faithfulness: an explanation may be correct while failing to faithfully reflect an incorrect claim decision. Moreover, the necessity of explicit visual evidence for scoring and explanation generation remains unclear. For example, the paper lacks a controlled ablation that removes the evidence provided by GroundingDINO. I would consider raising my rating if these concerns were addressed.
Questions for authors
  1. Please see Weakness 2.
  2. Complete decomposition into atomic claims is important for this task. Could the authors clarify how they ensure that the resulting claims are consistent and contain no duplicates or omissions?
  3. Why is the final score the average of all claim scores, with each claim given the same weight? This may reduce the impact of critical errors. Moreover, claims that include one another, such as “dog,” “black dog,” and “two black dogs,” are allowed to coexist, but equal-weight averaging does not explicitly account for their overlapping information. How is this issue addressed?
  4. Could the authors clarify how much of the improvement from evidence-aware GDPO comes from the policy objective versus the causal loss? An ablation retaining only the causal loss would help disentangle their contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.