- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces FAITH-GVT, evaluating textual faithfulness, visual faithfulness, and cross-modal consistency in explicitly grounded reasoning through image, text, and region interventions, alongside predicted-region masking. Experiments cover 51,463 samples from five datasets and four VLMs. The results reveal clear dissociations among textual explanations, region localization, and behavioral sensitivity, but some metric definitions and intervention designs do not adequately support their interpretation as measures of faithfulness.
Critical strengths
-
Jointly examines textual explanations and explicit regional evidence claims across natural-image and document tasks, offering greater diagnostic value than evaluating answers or explanation fluency alone.
-
Provides large-scale experiments with flip rates, target-match rates, spatial changes, and reasoning-format comparisons on common answer-flipped samples, supplying substantial data for further analysis.
Critical weaknesses
-
equates “the explanation is no longer supported after masking” with consistency, without first verifying support in the original image and within the predicted box. An explanation that was visually unsupported from the outset could receive a high score, while redundant evidence outside the box would be treated as inconsistency. The metric therefore does not reliably measure text–region compatibility.
-
The masking experiment changes both the image and the response requirements: C.5 prohibits inference and permits abstention, unlike the baseline prompt. also directly rewards self-reported unanswerability. Without controls using the original image under the same prompt and area-matched off-evidence masks, evidence removal cannot be separated from abstention tendencies or mask-area effects.
-
The random-region condition is not a clean control: C.1/C.2 add textual guidance to “use the highlighted region as a visual cue” only for random regions, not misleading regions. Random-box sizes are also matched to reference boxes rather than misleading boxes. Differences between these conditions therefore cannot be attributed solely to semantic targeting.
-
Answer flips do not establish that the model used the specified intervention content. On BoundingDocs, misleading-region target-match rates are only 7.44% for Qwen and 8.73% for InternVL, suggesting that many flips may reflect generic disruption rather than reliance on the intended alternative evidence. Interpreting low content-disclosure rates as undisclosed visual influence requires analysis on target-matched subsets and additional perturbation controls.
Questions for authors
Please refer to Weaknesses.