- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The authors look at VLMs and whether they can faithfully describe how their predictions depend on available and missing input modalities. - They propose an interventional evaluation protocol examining two complementary settings: retrospective attribution, which tests whether a model's stated reliance on individual modalities agrees with its behavior under modality removal, and prospective counterfactual self-knowledge, which evaluates whether a model can anticipate changes in its predictions when missing modalities are restored.
- Experiments involving ten VLMs across various benchmarks show that models systematically overestimate the sufficiency of their available evidence. Executed answer changes exceed predicted changes in 78 of 80 model–task–condition settings, while self-reported evidence insufficiency frequently fails to identify cases where restoring missing inputs changes predictions.
- The findings highlight limitations in using model-generated self-explanations as reliable indicators of modality dependence, motivating behavioral interventions as a means of evaluating explanation faithfulness.
Critical strengths
- Meaningful research problem with relevance to multimodal explainability and assurance.
- I think self-explanation faithfulness continues to be an important challenge underpinning the explainability and trustworthiness of contemporary generative AI systems.
- An alternative lens here looking at "modality missingness", where VLMs must operate under incomplete evidence, is useful and needed for the multimodal domain.
- The distinction between a model's stated evidential dependence and its actual behavior is particularly relevant in multimodal systems where the availability of information may change dynamically and on the fly.
- Useful interventional evaluation with informative empirical findings.
- The proposed protocol provides an executable way to interrogate whether a model's self-explanation agrees with its observed behavior under modality removal or restoration.
- The systematic underprediction of answer changes following modality restoration is particularly informative, and the authors' efforts to examine prompting and evaluation confounders strengthen the credibility of the reported discrepancies.
- Thoughtful experimental controls and transparency.
- I do appreciate the authors' efforts to examine the sensitivity of their findings to different prompting formulations and evaluation conditions. Example, in the matched-template comparisons that help address the concern that eliciting self-explanations may itself alter model behavior.
Critical weaknesses
- Limited novelty and explanatory depth relative to current established research
- The broader phenomenon of unreliable self-explanations, overestimated evidence sufficiency, and inconsistent responses to changing inputs is not entirely novel. Prior studies such as Parcalabescu and Frank (ICLR 2025) and Mayne et al. (ICML 2026) have already examined multimodal attribution, explanation faithfulness, and the predictive value of model self-explanations.
- I'm too sure of the clinical distinction here but it would appear to be focusing on prospective evaluation of model self-prediction when their modalities are removed or restored.
- While this is useful, it remains unclear what deeper scientific understanding this extension establishes beyond existing observations of self-explanation inconsistency.
- The proposed framework measures agreement between self-reported counterfactual predictions and executed model outputs. However, its interpretation as explanation faithfulness requires further consideration, particularly for probabilistic autoregressive models whose behavior may depend on decoding and input representation.
- Although the confounders are brought up and discussed, it remains unclear how the observed discrepancies relate to genuine modality dependence, mechanistic interpretability, or established multimodal information interactions such as redundancy, uniqueness, and synergy.
Lastly, the work remains primarily diagnostic, without sufficiently characterizing the underlying causes or offering an indicative methodology for improving self-explanation reliability. Are there any influence ideas on how we can improve self-reporting and robustness?
Questions for authors
- Can the authors distinct this work against existing multimodal attribution studies such as Parcalabescu et al and Mayne et al which was referenced in the paper? What additional scientific insight does the framework establish about the nature of self-explanation failures and can the authors provide a more defensible argument on this evaluation framework?
- Behavioral versus mechanistic faithfulness: The behaviour test is useful but it remains unclear how this criterion relates to the model's actual causal reliance on modality-specific information.
- In particular, identical outputs under modality ablation do not necessarily imply that the removed modality was unused, while changed outputs may arise from representation instability or cross-modal interference rather than genuine information insufficiency.
- Can the authors distinguish failures arising from genuinely missing task-relevant information from inconsistent utilization of redundant information or cross-modal interference?
- How might the proposed evaluation inform a practical methodology for improving reliability? Under real-world scenarios, where missing evidence may or may be restored under deployment, how do the authors see the calibration being actualised?
3 comments
Dear Reviewer L6XQ,
As the discussion period is about to close, we wanted to check whether our response addresses your concerns. In brief, we set our protocol against Parcalabescu & Frank and Mayne et al. and state the findings it adds; clarify what the behavioral criterion does and does not establish; show how the redundant and unique task regimes separate inconsistent use of equivalent inputs from genuinely missing information; and add a new analysis of causes with a mitigation strategy (Section 5.1) and a deployment procedure.
If this addresses your concerns, we would be grateful if you would consider updating your assessment. We welcome any further questions in the time remaining!
We would also like to express our appreciation for your thorough review! - this immensely helped us to enhance our paper.
Follow up response to rebuttals
- Thanks, the distinction from prior work is clearer, but it does seem to be an extension of existing follow up experiments in a very natural approach rather than a substantial novel insight.
- I can appreciate the new expt results on the models abilities to recognise available information, but this does not establish why the failure occurs. How do we distinguish between failures in self-reporting and failures arising from other effects such as reasoning?
- Prompting models and getting them aware of missing info provides improvements, but is it reliable? Or as with all probabalistic inference, it simply makes them more likely to report missing information just because I've "steered" it so. This practical significance remains unclear.
- IsoBench results show that answers can change even when the information is equivalent. So are they measuring genuine information insufficiency? or it is just an effect of inconsistent model behaviour.
Overall, the revisions strengthen some empirical findings, but I don't get a sense and understanding why these failures occur or how to reliably address them. I therefore maintain my original assessment.
Response to Reviewer L6XQ
We thank the reviewer for the careful and constructive review. The revised paper adds a new analysis of causes and mitigation (Section 5.1, Appendix A.8).
Relation to Parcalabescu & Frank and Mayne et al. (W1, Q1)
We agree that the unreliability of self-explanations is established, and we do not claim it as our finding. Our work differs in what is tested and against what:
Our framework adds four insights.
What the behavioral test establishes (W2, Q2, Q3)
We agree that identical outputs under ablation do not show that a modality was unused, and we make no claim about internal computation (footnote 1; Sec 5.2). The statements we test are themselves about outputs ("from the plot alone I would answer X"), so comparing them with the answer the model then gives is the matching test. This is what we mean by behavioral faithfulness.
Two points support the criterion. First, GoldenView gives an external anchor: for every model, accuracy is higher with the camera set that holds the annotated evidence, by a median of 27.9 pp. Second, decoding and prompts are controlled: decoding is greedy, run-to-run noise is one or two items in 41 (App. E.2), and the gap remains in 63 and 60 of 64 open-weight settings under a matched prompt template (App. A.3).
Missing information, redundant information and interference (W3, Q4)
Our task regimes follow the redundancy, uniqueness and synergy decomposition of Liang et al. (2023), and two of them fix the relation by construction.
Self-reports thus fail under both redundancy and uniqueness. For interference, the relevant case is a joint answer that matches neither single-input answer (BOTH in Table 2, the synergy-like case). Models understate it on all four tasks, and on max-flow it occurs on 10.9% of items although the inputs are equivalent. Our criterion does not separate interference from instability there; both count as unanticipated change.
Causes, mitigation and practical use (W4, Q5)
Our new analysis (Sec 5.1, App A.8) on Qwen3.5 27B & Gemma 4 31B and all four tasks locates the failure mainly in the self-report stage:
For deployment, this suggests:
On calibration: the protocol measures the two quantities to align, predicted and executed change, and the prompt above already narrows their gap by up to 12.7 pp (Tab. 24); full calibration is future work.
We hope this addresses your points and welcome further questions.