acceptodds
Under review as a conference paper at ICLR 2027

I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models

24 views

Abstract

Vision-language models (VLMs) are increasingly used in settings where some input modalities may be unavailable, yet we know little about whether they can faithfully explain how such missing information affects their own predictions. We introduce an interventional protocol for evaluating self-explanations of modality dynamics: models state what each modality alone would support, whether restoring a missing modality would change their answer, and whether the available evidence is sufficient; we then execute the corresponding intervention and compare these claims with realized behavior. We evaluate ten VLMs spanning open-weight and proprietary models across four tasks covering mixed, redundant, and unique modality regimes. We find a systematic tendency to overstate the sufficiency of available evidence. Models substantially underestimate the effect of restoring missing modalities: executed change exceeds predicted change in 78 of 80 model–task–condition settings, with task-level median executed change rates reaching 70.1% while median predicted rates remain at most 9.6%. Insufficiency claims have low recall, leaving many cases in which behavior changes despite a stated claim of sufficiency. Retrospective self-explanations show the same tendency, over-crediting single-input sufficiency in mixed regimes and interchangeability in redundant ones. Together, these results show that VLMs systematically mischaracterize how their predictions depend on available and missing evidence, motivating executable interventions as a behavioral test of multimodal self-explanations.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, faithfulness, self-explanation, missing modalities

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L6XQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The authors look at VLMs and whether they can faithfully describe how their predictions depend on available and missing input modalities. - They propose an interventional evaluation protocol examining two complementary settings: retrospective attribution, which tests whether a model's stated reliance on individual modalities agrees with its behavior under modality removal, and prospective counterfactual self-knowledge, which evaluates whether a model can anticipate changes in its predictions when missing modalities are restored.
  • Experiments involving ten VLMs across various benchmarks show that models systematically overestimate the sufficiency of their available evidence. Executed answer changes exceed predicted changes in 78 of 80 model–task–condition settings, while self-reported evidence insufficiency frequently fails to identify cases where restoring missing inputs changes predictions.
  • The findings highlight limitations in using model-generated self-explanations as reliable indicators of modality dependence, motivating behavioral interventions as a means of evaluating explanation faithfulness.
Critical strengths
  1. Meaningful research problem with relevance to multimodal explainability and assurance.
  • I think self-explanation faithfulness continues to be an important challenge underpinning the explainability and trustworthiness of contemporary generative AI systems.
  • An alternative lens here looking at "modality missingness", where VLMs must operate under incomplete evidence, is useful and needed for the multimodal domain.
  • The distinction between a model's stated evidential dependence and its actual behavior is particularly relevant in multimodal systems where the availability of information may change dynamically and on the fly.
  1. Useful interventional evaluation with informative empirical findings.
  • The proposed protocol provides an executable way to interrogate whether a model's self-explanation agrees with its observed behavior under modality removal or restoration.
  • The systematic underprediction of answer changes following modality restoration is particularly informative, and the authors' efforts to examine prompting and evaluation confounders strengthen the credibility of the reported discrepancies.
  1. Thoughtful experimental controls and transparency.
  • I do appreciate the authors' efforts to examine the sensitivity of their findings to different prompting formulations and evaluation conditions. Example, in the matched-template comparisons that help address the concern that eliciting self-explanations may itself alter model behavior.
Critical weaknesses
  1. Limited novelty and explanatory depth relative to current established research
  • The broader phenomenon of unreliable self-explanations, overestimated evidence sufficiency, and inconsistent responses to changing inputs is not entirely novel. Prior studies such as Parcalabescu and Frank (ICLR 2025) and Mayne et al. (ICML 2026) have already examined multimodal attribution, explanation faithfulness, and the predictive value of model self-explanations.
  • I'm too sure of the clinical distinction here but it would appear to be focusing on prospective evaluation of model self-prediction when their modalities are removed or restored.
  • While this is useful, it remains unclear what deeper scientific understanding this extension establishes beyond existing observations of self-explanation inconsistency.
  • The proposed framework measures agreement between self-reported counterfactual predictions and executed model outputs. However, its interpretation as explanation faithfulness requires further consideration, particularly for probabilistic autoregressive models whose behavior may depend on decoding and input representation.
  • Although the confounders are brought up and discussed, it remains unclear how the observed discrepancies relate to genuine modality dependence, mechanistic interpretability, or established multimodal information interactions such as redundancy, uniqueness, and synergy.

Lastly, the work remains primarily diagnostic, without sufficiently characterizing the underlying causes or offering an indicative methodology for improving self-explanation reliability. Are there any influence ideas on how we can improve self-reporting and robustness?

Questions for authors
  • Can the authors distinct this work against existing multimodal attribution studies such as Parcalabescu et al and Mayne et al which was referenced in the paper? What additional scientific insight does the framework establish about the nature of self-explanation failures and can the authors provide a more defensible argument on this evaluation framework?
  • Behavioral versus mechanistic faithfulness: The behaviour test is useful but it remains unclear how this criterion relates to the model's actual causal reliance on modality-specific information.
  • In particular, identical outputs under modality ablation do not necessarily imply that the removed modality was unused, while changed outputs may arise from representation instability or cross-modal interference rather than genuine information insufficiency.
  • Can the authors distinguish failures arising from genuinely missing task-relevant information from inconsistent utilization of redundant information or cross-modal interference?
  • How might the proposed evaluation inform a practical methodology for improving reliability? Under real-world scenarios, where missing evidence may or may be restored under deployment, how do the authors see the calibration being actualised?
3 comments
AuthorsOpenReview ↗

Response to Reviewer L6XQ

We thank the reviewer for the careful and constructive review. The revised paper adds a new analysis of causes and mitigation (Section 5.1, Appendix A.8).

Relation to Parcalabescu & Frank and Mayne et al. (W1, Q1)

We agree that the unreliability of self-explanations is established, and we do not claim it as our finding. Our work differs in what is tested and against what:

Parcalabescu & Frank (2025)Mayne et al. (2026)Ours
TestsHow much image and text contribute to the answer and to the explanation (Shapley values)Whether an explanation helps a second model predict behaviorEach statement a model makes about its reliance on a modality
AgainstEach otherThe second model's accuracyThe same model's own answers after the stated change of input
YieldsA similarity scoreAn average predictive gainA check per statement and item

Our framework adds four insights.

  1. The error is one-sided. Models predict less change than occurs in 78 of 80 settings, and never more in the main grid.
  2. The signal exists but is under-produced. Insufficiency flags are precise (median precision 77.6% on MM-IMDb, 100% on GoldenView) but miss many cases that change (median recall 13.9% and 58.6%). This sharpens the positive result of Mayne et al.: the information is there, but models do not reliably volunteer it.
  3. The error depends on how the modalities relate (see Q4).
  4. The failure is not mainly perceptual, and a simple prompt reduces it (see Q5).

What the behavioral test establishes (W2, Q2, Q3)

We agree that identical outputs under ablation do not show that a modality was unused, and we make no claim about internal computation (footnote 1; Sec 5.2). The statements we test are themselves about outputs ("from the plot alone I would answer X"), so comparing them with the answer the model then gives is the matching test. This is what we mean by behavioral faithfulness.

Two points support the criterion. First, GoldenView gives an external anchor: for every model, accuracy is higher with the camera set that holds the annotated evidence, by a median of 27.9 pp. Second, decoding and prompts are controlled: decoding is greedy, run-to-run noise is one or two items in 41 (App. E.2), and the gap remains in 63 and 60 of 64 open-weight settings under a matched prompt template (App. A.3).

Missing information, redundant information and interference (W3, Q4)

Our task regimes follow the redundancy, uniqueness and synergy decomposition of Liang et al. (2023), and two of them fix the relation by construction.

RegimeA changed answer meansWhat we observe
Redundant (IsoBench): both inputs carry the same informationNot missing information: equivalent inputs are used inconsistentlyAnswers change on a median of 7.4% (parity) and 34.4% (max-flow) of inputs. Models predict 0% and almost never flag it
Unique (GoldenView): one camera set holds the annotated evidenceMissing information, when that set is withheldEven when a model recognises that the evidence camera is absent, it still calls its input sufficient in 42% (Qwen3.5 27B) and 91% (Gemma 4 31B) of cases (set B; n = 19 and 11; Table 26)
Mixed (MM-IMDb)Either; not separable per itemThe largest gap: 70.1% executed against 9.6% predicted

Self-reports thus fail under both redundancy and uniqueness. For interference, the relevant case is a joint answer that matches neither single-input answer (BOTH in Table 2, the synergy-like case). Models understate it on all four tasks, and on max-flow it occurs on 10.9% of items although the inputs are equivalent. Our criterion does not separate interference from instability there; both count as unanticipated change.

Causes, mitigation and practical use (W4, Q5)

Our new analysis (Sec 5.1, App A.8) on Qwen3.5 27B & Gemma 4 31B and all four tasks locates the failure mainly in the self-report stage:

  • Models can read their inputs. Probe accuracy on input content is 99–100% on MM-IMDb, 78–100% on IsoBench and 60–94% on GoldenView.
  • Restating the input does not help. Listing its facts first gives no significant gain.
  • Asking what is missing does. When the model first lists what its input leaves open, detection of insufficient input rises by 5.0–17.4 pp on every MM-IMDb cell and by 25.0-61.9 pp on GoldenView, without any training.

For deployment, this suggests:

  1. Before the model answers, ask what its input leaves open.
  2. Act on INSUFFICIENT claims, which are precise. Treat SUFFICIENT claims as unverified.
  3. Verify them by restoring the missing input where possible; otherwise, estimate their error rate offline with our protocol.

On calibration: the protocol measures the two quantities to align, predicted and executed change, and the prompt above already narrows their gap by up to 12.7 pp (Tab. 24); full calibration is future work.

We hope this addresses your points and welcome further questions.

Dear Reviewer L6XQ,

As the discussion period is about to close, we wanted to check whether our response addresses your concerns. In brief, we set our protocol against Parcalabescu & Frank and Mayne et al. and state the findings it adds; clarify what the behavioral criterion does and does not establish; show how the redundant and unique task regimes separate inconsistent use of equivalent inputs from genuinely missing information; and add a new analysis of causes with a mitigation strategy (Section 5.1) and a deployment procedure.

If this addresses your concerns, we would be grateful if you would consider updating your assessment. We welcome any further questions in the time remaining!

We would also like to express our appreciation for your thorough review! - this immensely helped us to enhance our paper.

Reviewer_L6XQOpenReview ↗

Follow up response to rebuttals

  • Thanks, the distinction from prior work is clearer, but it does seem to be an extension of existing follow up experiments in a very natural approach rather than a substantial novel insight.
  • I can appreciate the new expt results on the models abilities to recognise available information, but this does not establish why the failure occurs. How do we distinguish between failures in self-reporting and failures arising from other effects such as reasoning?
  • Prompting models and getting them aware of missing info provides improvements, but is it reliable? Or as with all probabalistic inference, it simply makes them more likely to report missing information just because I've "steered" it so. This practical significance remains unclear.
  • IsoBench results show that answers can change even when the information is equivalent. So are they measuring genuine information insufficiency? or it is just an effect of inconsistent model behaviour.

Overall, the revisions strengthen some empirical findings, but I don't get a sense and understanding why these failures occur or how to reliably address them. I therefore maintain my original assessment.

Reviewer_ohr1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether VLMs can faithfully characterize their own dependence on different input modalities, particularly when some modalities are missing. The authors propose an intervention-based evaluation framework consisting of two complementary settings: retrospective attribution, which compares models' self-reported modality dependence with their actual behavior under modality ablation, and prospective counterfactual self-knowledge, which evaluates whether models can anticipate changes in their predictions when a missing modality is restored. The evaluation covers ten VLMs across four tasks with different modality relationships. Results show that models systematically underestimate the impact of restoring missing modalities and tend to overestimate the sufficiency of available evidence.

Critical strengths
  1. Important research question and a meaningful evaluation framework. The paper investigates whether VLMs can accurately explain their own modality dependence, going beyond conventional missing-modality robustness studies. The proposed intervention-based framework provides a practical way to evaluate behavioral faithfulness through retrospective and prospective assessments.

  2. Comprehensive experimental evaluation. The experiments span ten VLMs and four tasks with different modality relationships, revealing a consistent tendency to underestimate answer changes. Additional robustness analyses, including prompt ablations, matched-prompt comparisons, and alternative answer-matching criteria, further support the main findings.

Critical weaknesses
  1. Limited investigation into underlying mechanisms and potential mitigation strategies. While the paper identifies systematic discrepancies between VLMs' self-explanations and their actual behavior, it provides limited insights into why these failures occur. A deeper investigation into the underlying mechanisms, such as modality processing limitations, cross-modal interactions, or self-explanation biases, would strengthen the contribution. Furthermore, exploring potential mitigation strategies could help establish the practical implications of the findings beyond identifying the phenomenon.

  2. Limited validation of the stability of the proposed diagnostic framework. The paper evaluates self-explanation faithfulness by comparing predicted and actual answer changes under modality interventions. However, the diagnostic measurements exhibit noticeable variations across different prompts and datasets. For example, Appendix A.4 shows that removing the copy-answer instruction increases the median predicted-change rate on mm-IMDb from 10.3% to 32.0%, while the magnitude of the predicted–executed change gap also varies substantially across datasets. Although the overall trend remains consistent, further analysis is needed to establish the stability and comparability of the proposed diagnostic framework across different evaluation settings.

Questions for authors

Q1. Could the authors provide further insights into the potential causes of the observed self-explanation failures? Are there any preliminary findings on strategies that could mitigate these failures?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ohr1

We thank the reviewer for the thoughtful and constructive review. The revised paper addresses your first point with a new analysis (Section 5.1, Appendix A.8). We respond to both points below.

Causes and mitigation (W1, Q1)

In short: the failures do not come from reading the inputs, but from the self-report itself, which accounts for what the model has and not for what it lacks. Asking the model what its input leaves open reduces, but does not close, the gap.

We went beyond identifying the phenomenon and asked two questions: where do these failures originate, and can they be reduced at inference time, without any training? We ran the analysis on Qwen3.5 27B and Gemma 4 31B, on all four tasks and in both settings.

QuestionWhat we didResult
Can the models read their inputs?Multiple-choice questions about the content of each input, built from dataset ground truth99–100% correct on MM-IMDb (chance 25%), 78–100% on IsoBench (chance 25–50%) and 60–94% on GoldenView (chance 17–25%)
Does it help if the model spells out what its input contains?The model first lists the facts in its input, in the same call or in a separate callNo significant gain in any cell
Does it help to ask what the input leaves open?In a first call, the model also lists what its input does not settle and what the missing input could addThe share of answer-changing cases that the model flags as insufficient rises by 5.0–17.4 pp in all four MM-IMDb cells, and by 25.0–61.9 pp in three of four GoldenView cells (the fourth was already at 100%)

A cell is one model on one task with one input withheld.

This tells us two things about the causes you list. First, modality processing is not the main limitation: the models read their inputs well, and having them restate what they read does not help. Second, the failure lies in the self-report itself, which accounts for what the model has and not for what it lacks. GoldenView shows this directly. On questions where the model itself correctly states that the camera holding the evidence is absent, it still calls its input sufficient in 91% of cases (Gemma 4 31B) and 42% (Qwen3.5 27B). Once it is asked what is missing, these fall to 27% and 11% (camera set B; 11 and 19 questions).

This gives a promising mitigation strategy that needs no training, only prompting: ask the model what its input leaves open before it answers. Beyond the gains in the table, it raises predicted change significantly in six of the eight MM-IMDb and GoldenView cells and lowers silent sufficiency failures by up to 20 pp. When both inputs are given, it reduces false credit to a single input on MM-IMDb for both models. On IsoBench, it makes models more likely to withhold credit from an input that does not reproduce their answer, by up to 30.8 pp.

Stability of the diagnostic (W2)

The measured levels do move with the prompt and the dataset. The direction of the result does not: models predict less change than occurs.

What we variedWhereCells in which executed change exceeds predicted change
Models, tasks and withheld input (main result)Table 378 of 80
Wording of the self-report promptAppendix A.452 of 64; lower in 4, by at most 6.9 pp
Same prompt template on both sidesAppendix A.363 and 60 of 64
Looser answer matching; thinking enabledAppendices A.6, A.516 of 16 each

On your example: without the copy-answer instruction, predicted change on MM-IMDb rises to 32.0%, against 68.8% executed on the same cells. The size of the gap differs across datasets as expected, since the tasks differ in how much the missing input matters (we have constructed and picked datasets so that we cover different natures of multimodal interactions, e.g. isomorphism, uniqueness, and synergy). We therefore read the diagnostic by its direction and within a task, and do not compare absolute levels across tasks.

Many thanks for your constructive feedback, which led to the new analysis and made the paper stronger. We hope we have addressed your points, and we very much look forward to answering any further questions.

Dear Reviewer ohr1,

As the discussion period is about to close, we wanted to check whether our response addresses your concerns. Following your suggestion, the revised paper adds an analysis of causes and mitigation (Section 5.1, Appendix A.8): the models read their inputs well, the failure lies in the self-report, and asking the model what its input leaves open raises insufficiency recall by up to 62 pp without any training. Our response also shows that the direction of the main result holds across the prompt, template, matching and thinking variants we tested.

If this resolves your concerns, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

Reviewer_EuJ4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces an evaluation protocol to test whether VLMs can explain their modality dependency. The test includes two settings. First, whether the model can explain which modality it used. Second, whether the model can explain how its prediction will change if the missing modality is restored. The self-report is elicited by prompting and compared with the model’s behavior when the modality is actually removed or restored. The paper tested this on 10 VLMs using four tasks from three benchmarks.

Critical strengths
  • The paper is well motivated and addresses an important problem. Faithfulness of multimodal self-explanations is an important question.
  • The proposed protocol is inference-only, so it can be applied to both open-weight and proprietary models.
Critical weaknesses
  • Presentation. Figure 1 renders poorly in Chrome. Many sentences, especially in the method section, are hard to read. For example, lines 186–188, “Give the model both inputs. Its report names the answers it claims each input alone would have supported, r_AB = (ŷ_A, ŷ_B); the single-input plain arms supply the same two answers behaviorally.” does not read like a method description in a paper. The writing should be substantially improved. Some symbols are poorly defined. For example, the term “arm” is used throughout without definition, and y, ŷ and ỹ are introduced only briefly in one paragraph of Section 3.1, making them easy to confuse in later sections. A simple figure with notations and actual examples could make it easier to follow.

  • Missing baseline comparison. Has this protocol been empirically compared to other existing protocols? How do they differ, and what new finding did this protocol discover that others could not? I recommend the authors compare and discuss this more.

Questions for authors
  • Could the authors revise the method section for clarity? What are the input and output examples in the two settings, and how are they compared? I found the exact prompts in the appendix, but pulling a summarized version into the main figure would make it much clearer.
  • Could the authors elaborate on an empirical comparison with existing protocols?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EuJ4

We thank the reviewer for the careful reading and the constructive feedback. We have uploaded a revised version of the paper and respond to each point below.

Presentation (W1, Q1)

We have rewritten the all the parts and especially those you pointed to. The method section now reads as a conventional method description, including the passage at lines 186 to 188 of the original. Section 3.1 defines "condition" and "arm" where they first appear, and states in one sentence what y, ỹ and ŷ each stand for: the model's answer, its answer when it is also asked to explain itself, and the answer it claims it would give under a different set of inputs.

On Figure 1: thank you for flagging this. We traced it to transparency layers left when we prepared it in Photoshop, which Chrome's PDF viewer displays inconsistently. We fixed the issue. The figure lays out how the evaluation is run end to end in both settings: what the model is given, what it reports, which intervention we then carry out, and what is compared.

As a concrete example, take MM-IMDb genre prediction, where the inputs are a film's plot summary and its poster:

  • Setting 1 (both inputs). Besides its genres, the model states the genres it would have given from the plot alone (PLOT_ALONE) and from the poster alone (POSTER_ALONE). We then run it on each input alone and compare.
  • Setting 2 (one input). Besides its genres, the model states the genres it would give with the poster added (WITH_POSTER) and marks its input SUFFICIENT or INSUFFICIENT. We then run it on both inputs and check whether its answer changed.

Appendix C gives every prompt in full, and Appendix C.6 shows a real input for each task.

Comparison with existing protocols (W2, Q2)

To our knowledge, no existing protocol tests whether a model's own account of how it depends on each modality holds up when a modality is missing. There is therefore no existing protocol we could run alongside ours on this question. Providing one is the contribution of the paper.

For completeness, we hereby present the following table that sets our protocol next to the two closest works.

Parcalabescu & Frank (ICLR 2025)Mayne et al. (ICML 2026)Ours
QuestionHow much do image and text each contribute to a model's answer and to its explanation?Does a model's explanation help predict how it will behave on related inputs?Is what a model says about its reliance on each modality borne out by what it does?
What is testedContribution scores computed for each input token and image patch (Shapley values)Free-form explanationsExplicit statements by the model: the answer it would give from each input alone, the answer it would give with the missing input added, and whether its input is sufficient
Tested againstThe scores for the answer, compared with the scores for the explanationA second model's predictions of the first model's answersThe same model's own answers once the stated change of input is carried out
What the result isA similarity score between two sets of contributionsAn average gain in how well the model can be predictedA check of each statement on each item: it held or it did not
SettingImage and text both present; 3 open-weight vision-language modelsText only; 18 language modelsOne modality missing, and both present; 10 VLMs, 2 of them proprietary

The two works are complementary to ours. Parcalabescu & Frank measure how much each modality contributes. Mayne et al. show that explanations carry useful information on average. Neither tests, item by item, whether what a model says about its own inputs is correct, and neither considers a missing modality. That test is what we contribute. It is also what produced our main findings: models predict far less change than occurs when a missing modality is restored (in 78 of 80 settings), and they rarely flag their input as insufficient even when restoring the other input changes their answer (median recall 13.9% on MM-IMDb).

Additional analysis: Causes and Mitigation

To strengthen the paper further, and in the hope that it also speaks to your second question, we went beyond identifying the phenomenon. We examined possible causes of these failures and tested mitigation strategies, which also shows how the findings can be put to practical use. Section 5.1 describes this analysis, with full results in Appendix A.8. In short: the models can read their inputs, so perception alone does not explain the failures. We also identify a promising mitigation strategy: asking the model to state what its input leaves open before it answers. This raises its detection of insufficient input by up to 17 percentage points on MM-IMDb and by up to 62 points on GoldenView.

Many thanks for your constructive feedback and for your efforts to make our work better. We hope we have addressed all your points, and we very much look forward to answering any further questions.

Dear Reviewer EuJ4,

As the discussion period is about to close, we wanted to check whether our response and the revised paper address your concerns. In brief: we rewrote the method section, defined "condition", "arm" and the notation at first use, fixed the Figure 1 rendering issue in Chrome, walked through both settings with a concrete example, and set our protocol against the two closest existing works. We also added a new analysis of causes and mitigation (Section 5.1).

If these changes resolve your concerns, we would be grateful if you would consider updating your score. We are happy to clarify anything further in the time remaining.

I appreciate the authors' response. I now better understand how the paper differs from existing works. However, I am not fully convinced that this difference in method leads to a substantially new finding. Therefore, I keep my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

We thank all three reviewers for their time and constructive feedback. Below we summarize the reviews, our responses, and the changes in the revised paper.

Strengths noted by the reviewers

  • An important and well-motivated problem: the faithfulness of multimodal self-explanations under missing modalities (EuJ4, ohr1, L6XQ).
  • An executable protocol (L6XQ) that is inference-only and so applies to open-weight and proprietary models alike (EuJ4).
  • A comprehensive evaluation across ten VLMs and four tasks (ohr1), with robustness analyses and controls that strengthen the credibility of the findings (ohr1, L6XQ).
  • The systematic underprediction of answer changes, which L6XQ found particularly informative.

All three reviewers rated soundness as having minor or fixable concerns. The concerns raised, and how we addressed them:

1. Presentation (EuJ4). We rewrote the method section, defined "condition", "arm" and the notation at first use, and fixed the Figure 1 rendering issue in Chrome. Our response walks through both settings with a concrete example.

2. Relation to prior work (EuJ4, L6XQ). To our knowledge, no existing protocol tests whether a model's own account of its modality dependence holds up when a modality is missing. Parcalabescu & Frank (2025) measure how much each modality contributes, and Mayne et al. (2026) show that explanations are predictive on average. We test each statement a model makes against its own behavior under the stated intervention. This shows that the error is one-sided (78 of 80 settings) and that insufficiency claims are precise but under-produced.

3. Causes and mitigation (ohr1, L6XQ). New Section 5.1 and Appendix A.8, on Qwen3.5 27B and Gemma 4 31B across all four tasks. The models identify the content of their inputs well above chance, and restating it does not help, so the failure is not mainly perceptual. It lies in the self-report, which accounts for what the model has but not for what it lacks. Asking the model what its input leaves open raises insufficiency recall by up to 62 pp without training, which gives a practical procedure for deployment.

4. Stability of the diagnostic (ohr1). Levels move with the prompt and with the dataset, the latter by design, but the direction does not. Executed change exceeds predicted change in 52 of 64 cells across alternative prompt wordings (never lower by more than 6.9 pp), in 63 and 60 of 64 under matched templates, and in 16 of 16 both with a looser matching rule on MM-IMDb and with thinking enabled.

5. Behavioral criterion and modality interactions (L6XQ). We test behavioral faithfulness, matched to self-reports that are themselves claims about outputs, and make no mechanistic claim. Our task regimes separate the redundant case (IsoBench), where a changed answer cannot reflect missing information, from the unique case (GoldenView), where withholding the annotated camera set removes decisive evidence. Self-reports fail in both.

Changes in the revised paper

  • Method section (Section 3) rewritten, with "condition", "arm" and the notation defined at first use.
  • Figure 1 re-exported to fix the rendering issue.
  • New analysis of causes and mitigation (Section 5.1, Appendix A.8), reflected in the abstract and contributions.

We hope the revision and our responses address the reviewers' concerns, and we remain available for any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.