acceptodds
Under review as a conference paper at ICLR 2027

What You See Is Not What Matters: Attacking VLM Explanation Faithfulness

Abstract

Visual explanations are increasingly used to understand the decisions of Vision-Language Models (VLMs), yet their robustness under adversarial perturbations remains poorly understood. Existing studies mainly focus on saliency changes, but a substantial explanation shift does not necessarily imply unfaithfulness. We investigate whether prediction-preserving perturbations can redirect saliency toward regions that provide weaker support for the model's prediction. To support this evaluation, we propose the Faithfulness-Oriented Adversarial (FOA) Attack, which uses explanation faithfulness as an optimization signal to stress-test highlighted regions. By detaching saliency and mask generation, FOA avoids backpropagation through the explainer while enabling faithfulness-oriented updates. Across five VLM explanation methods and three datasets, our evaluation shows that faithfulness can be substantially degraded while preserving the original prediction, with FOA consistently revealing stronger vulnerabilities than X-Shift, a state-of-the-art attack on VLM explanations that primarily targets saliency shifts rather than explanation faithfulness. FOA perturbations also transfer effectively across explainers, suggesting that these vulnerabilities extend across explanation methods. These results demonstrate that highlighted regions can become less representative of the evidence supporting a prediction even when the prediction is unchanged. Our findings show that explanation robustness should be assessed not only by where explanations shift, but also by whether the resulting explanations remain faithful.

Primary area:
interpretability and explainable AI
Keywords:
Adversarial Robustness, Explainable AI, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n3HfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Faithfulness-Oriented Adversarial (FOA) attack, which introduces prediction-preserving perturbations that can redirect saliency toward regions that provide weaker support for the VLM's predictions. Specifically, FOA uses insertion- and deletion-based objectives with a stop-gradient strategy to avoid backpropagation through the explanation method. Across five explanation methods and three datasets, the attack can substantially reduce faithfulness across multiple explainers while preserving the original predictions. This work highlights that a substantial explanation shift does not necessarily imply unfaithfulness and that we should assess explanations both in terms of where they shift and whether the resulting explanations remain faithful.

Critical strengths

Strength 1: The paper is well-motivated and has a clean formulation of the research problem. The authors observe and highlight that a change in the saliency map does not mean that the resulting explanation is unfaithful. As a result, the authors move beyond measuring how much an explanation changes and instead investigate whether the highlighted regions remain informative for the model's prediction.

Strength 2: The proposed FOA framework provides a simple way to incorporate established faithfulness metrics into adversarial optimization. In particular, the stop-gradient strategy avoids significant computational complexity and cost associated with differentiating through the explanation method.

Strength 3: The experimental evaluation covers multiple representative explanation methods and multiple datasets, ensuring that the proposed attack is not restricted to a single explanation technique or dataset.

Critical weaknesses

Weakness 1: Although the paper shows that the FOA attack induces changes in saliency maps and degrades explanation faithfulness metrics, it remains somewhat unclear how much of the observed degradation is attributable to changes in saliency rankings versus changes in the model's response to masked inputs. Since the attack directly optimizes the model's confidence on masked images, additional control experiments could help disentangle these two effects. For example, evaluating the clean and adversarial saliency maps on the same adversarial image using IMD and ROAD could be useful. This additional analysis could strengthen the paper.

Weakness 2: Currently, all experiments use the same underlying ViT-B/16 CLIP model (as mentioned in Section 4.1, page 6). It remains unclear whether the proposed attack would exhibit similar effectiveness on larger CLIP variants or alternative VLM architectures. Additional experiments on other backbones could help strengthen the paper.

Weakness 3: While I appreciate that the authors propose gradient detachment through the (M_t) generation branch to save computation and potentially avoid second-order derivatives in some cases, the performance impact of using the surrogate gradient in Equation (8) remains unclear.

Questions for authors

Question 1: Can the authors potentially provide additional analysis on the sources of faithfulness degradation under attack? For example, a quick controlled study that evaluates the clean and adversarial saliency maps on the same adversarial image could be helpful.

Question 2: Can the authors perform similar experiments on different VLM backbones (a larger one or a different VLM)?

Question 3: Can the authors explain how they chose the hyperparameters in Section 4.1, particularly the step size (α=4/255)(\alpha=4/255) and the number of optimization iterations (K=50)? Are the experimental results consistent across different step sizes and numbers of optimization iterations?

Question 4: Can the paper also report the prediction accuracy after the attack? The main idea is that we can introduce attacks without reducing accuracy, so it would be useful to verify how the FOA attack achieves both objectives.

5 comments
AuthorsOpenReview ↗

Weakness 1 and Question 1 - Controlled Ablation on Saliency Ranking and Masked-Input Responses

Weakness 1

Weakness 1: Although the paper shows that the FOA attack induces changes in saliency maps and degrades explanation faithfulness metrics, it remains somewhat unclear how much of the observed degradation is attributable to changes in saliency rankings versus changes in the model's response to masked inputs. Since the attack directly optimizes the model's confidence on masked images, additional control experiments could help disentangle these two effects. For example, evaluating the clean and adversarial saliency maps on the same adversarial image using IMD and ROAD could be useful. This additional analysis could strengthen the paper.

Quesiton 1

Question 1: Can the authors potentially provide additional analysis on the sources of faithfulness degradation under attack? For example, a quick controlled study that evaluates the clean and adversarial saliency maps on the same adversarial image could be helpful.

We thank the reviewer for this valuable suggestion. Following the reviewer's recommendation, we conducted an additional experiment to evaluate clean and adversarial saliency maps on the same adversarial images, using the same model and masking protocol. The results are presented in the following table:

Table 5. Faithfulness evaluation of clean and adversarial saliency maps on the same adversarial images generated by FOA. Higher IMD (↑\uparrow), and lower Del$_{\text{mean}}$ (↓\downarrow) and ROAD (↓\downarrow), indicate the saliency map is more faithful. Results are averaged across the three FOA variants (IOA, DOA, and IOA+DOA) and the three datasets. The Without Attack rows report the faithfulness performance of each explainer under clean conditions, averaged across the three datasets.

ExplainerSaliency MapIMD ↑Del$_{\text{mean}}$ ↓ROAD ↓
GradECLIPWithout Attack0.61280.13940.2100
Clean Saliency0.26270.28000.2586
Adversarial Saliency0.32830.20080.2688
GAMEWithout Attack0.61280.13940.2100
Clean Saliency0.23810.27040.2470
Adversarial Saliency0.30720.22760.2743
Grad-CAMWithout Attack0.36240.22710.2139
Clean Saliency0.04200.31840.3143
Adversarial Saliency0.09560.31030.2895
MaskCLIPWithout Attack0.55580.14710.2121
Clean Saliency0.24980.26730.2486
Adversarial Saliency0.31910.20010.2590
RISEWithout Attack0.33520.27820.2686
Clean Saliency0.24980.26730.2486
Adversarial Saliency0.16520.33970.3107

However, we think this comparison is not entirely appropriate because the clean and adversarial saliency maps are generated on different input images from those they were originally generated to explain, resulting in a mismatch between the saliency maps and their corresponding inputs.

Motivated by this limitation, we therefore designed an alternative ablation study. Specifically, we compared three masking strategies: (1) Adaptive-mask, which dynamically updates the saliency ranking during attack optimization, following the original FOA; (2) Clean-mask, which uses a fixed saliency ranking generated from the clean image; and (3) Random-mask, which uses a fixed randomly generated ranking. This comparison allows us to examine whether dynamically adapting the salient regions provides additional attack effectiveness beyond changes in the model's responses to masked inputs.

We refer the reviewer to our response to Reviewer JxgH regarding Weakness 1 and Question 1 for a detailed discussion of this ablation study (see https://openreview.net/forum?id=sQeTssIaEL&noteId=QFew3CP4ds).

The results, reported in Table 4 , Appendix D, show that Adaptive-mask generally causes greater faithfulness degradation than Clean-mask and Random-mask under both Del$_{\text{mean}}$ and ROAD, particularly for IOA and DOA. Meanwhile, Clean-mask also produces non-negligible degradation, suggesting that changes in model responses to masked inputs may also contribute to FOA's effectiveness even without adaptive saliency ranking.

AuthorsOpenReview ↗

Weakness 2 and Question 2 - Attack Across Different VLM Backbones (BLIP and SigLIP)

Weakness 2

Weakness 2: Currently, all experiments use the same underlying ViT-B/16 CLIP model (as mentioned in Section 4.1, page 6). It remains unclear whether the proposed attack would exhibit similar effectiveness on larger CLIP variants or alternative VLM architectures. Additional experiments on other backbones could help strengthen the paper.

Question 2

Question 2: Can the authors perform similar experiments on different VLM backbones (a larger one or a different VLM)?

We thank the reviewer for this suggestion. To investigate whether our findings generalize beyond CLIP, we have extended our experiments to two additional VLM backbones, BLIP and SigLIP.

We have addressed this weakness and Question in our response to Reviewer JxgH (see Question 3: https://openreview.net/forum?id=sQeTssIaEL&noteId=FgWmeKDxG7). We kindly refer the reviewers to that response for the detailed results and discussion.

The results, reported in Appendix F - Table 6, show that FOA also effectively degrades explanation faithfulness on these backbones. These findings provide further evidence that the vulnerabilities exposed by FOA are not specific to the CLIP architecture.

AuthorsOpenReview ↗

Question 4 - Prediction Accuracy

Question 4

Question 4: Can the paper also report the prediction accuracy after the attack? The main idea is that we can introduce attacks without reducing accuracy, so it would be useful to verify how the FOA attack achieves both objectives.

We thank the reviewer for this suggestion. FOA explicitly preserves the original predicted class through its candidate-selection mechanism: only candidates with unchanged predicted labels are accepted, and the one with the lowest attack objective is retained (Line 27 of Algorithm 1 - Appendix A). Moreover, the KL-divergence regularization further encourages prediction distributions to remain close to the original. Since our evaluation set contains correctly classified samples, the accepted adversarial examples preserve the original classification accuracy.

Regarding the confidence scores, which may additionally help address your concern, we kindly refer the reviewers to our response to Reviewer JxgH (Question 3: https://openreview.net/forum?id=sQeTssIaEL&noteId=yfQ6evpfuV) for the detailed results and discussion.

Reviewer_6PgmOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper first makes the claim that adversarial robustness of visual explanations for VLMs is under-explored and the proposes an adversarial attack that does not change the model prediction but changes the saliency map of the model. The idea is that the saliency map gets diverted to relatively less important regions such that the final label prediction remains unchanged. The diversion mechanism is devised by getting the adversarial feedback from Insertion-Deletion metric. That is, this score gets hurt, and in the process the saliency gets corrupted, but the feedback retains the model prediction loss as well to keep the original prediction (label) intact. The approach is evaluated against a method Xshift, showing favorable results - in the sense that the saliency map gets corrupted.

Critical strengths

The direction of VLM saliency behavior is relatively less explored and the paper explores this direction. Disentanglement between saliency and prediction becomes more visible with such a study.

Critical weaknesses

There are major weaknesses in this work: -The core objective of distorting saliency map while preserving original prediction does not make much sense. The proposed FOA is essentially an attack that does not change the prediction. This makes it questionable to frame the approach as an 'adversarial' attack. The approach sounds more like a probing tool to investigate correlation between saliency and prediction. However, the approach is not posed like that, which makes it hard to appreciate its utility.

  • The approach is based on directly optimizing for the Instertion-deletion metric, which is just one of the metrics and has it pros and cons. The success case is mostly made by presenting AUC for the metric that the approach is directly optimized for.
  • The claim that it is the first study to investigate the faithfulness of VLM explanations is objectionable in the presence of x-shift work.
  • The writing is often misleading, e.g., the very first sentence referring to approaches appearing in 2018 and 2017 as VLM explanation methods is technically incorrect.
  • Evaluations are not convincing. For benchmarking only x-shift is used. Whereas the authors note that this is currently the only method, in Tab 1, that method seems to have very poor results. Often distorting metric values worse than 'without attack'. Comparison against such a method does not make a convincing case, especially when AUC curves mainly compared with x-shift. Not to mention, the proposed approach is optimized against the insertion-deletion metric, hence its no surprise that its AUC curves are better. Also, 16/255 is a significant adversarial budget.
Questions for authors
  • Does the proposed attack change the output label of the model?
  • What is the main difference between the proposed method and x-shift?
  • What is the main utility of the proposed method?
  • How well does the attack work with 4 and 8/255 budget?
  • Does increasing the budget finally change the model prediction (label) as well?
10 comments
AuthorsOpenReview ↗

The Motivation and Adversarial Nature of Prediction-Preserving Explanation Manipulation

Reviewer's comment

There are major weaknesses in this work: -The core objective of distorting saliency map while preserving original prediction does not make much sense. The proposed FOA is essentially an attack that does not change the prediction. This makes it questionable to frame the approach as an 'adversarial' attack. The approach sounds more like a probing tool to investigate correlation between saliency and prediction. However, the approach is not posed like that, which makes it hard to appreciate its utility.

We respectfully disagree with the concern that an attack preserving the model's original prediction is inherently meaningless or should not be considered an adversarial attack.

First, regarding the meaningfulness of this problem, conventional adversarial attacks primarily target the prediction behavior of a model, for example, by inducing misclassification. However, the robustness of an AI system is not determined solely by whether its predicted label is correct. In systems where explanations are used for model auditing, decision support, or human oversight, the integrity of the explanation is also an important component of system reliability. An explanation that highlights misleading or irrelevant image regions while the model retains its original prediction can create a false impression of how the model arrives at its decision. Investigating this vulnerability is therefore meaningful, particularly for applications that rely on explanations to assess or interpret model decisions.

Besides, manipulating explanations while preserving model predictions is an established research direction in explanation robustness. Prior studies have explicitly investigated adversarial attacks on explanations, demonstrating that carefully crafted input perturbations can substantially alter model explanations without changing, or significantly affecting, the model's predictions. For examples:

  1. Interpretation of Neural Networks Is Fragile (AAAI 2019, https://arxiv.org/abs/1710.10547) demonstrates that adversarial perturbations can produce substantially different interpretations while preserving the predicted label.

  2. Explanations Can Be Manipulated and Geometry Is to Blame (NeurIPS 2019, https://arxiv.org/abs/1906.07983) shows that explanations can be manipulated to the predefined visual pattern while keeping the model's output approximately constant.

  3. Resilience of Bayesian Layer-Wise Explanations under Adversarial Attacks (IJCNN 2022, https://arxiv.org/abs/2102.11010) further demonstrates that saliency interpretations of deterministic neural networks can be substantially altered by adversarial input perturbations even when the attacks fail to change the predicted class,

  4. More recently, Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations (2026, https://arxiv.org/abs/2605.16651) (X-shift) has also explored explanation manipulation in vision-language models while preserving prediction outcomes.

  5. Furthermore, the importance of explanation manipulation has also motivated research on defensive mechanisms. For example, Defense Against Explanation Manipulation (Big Data 2022, https://arxiv.org/abs/2111.04303)

These studies establish prediction-preserving explanation manipulation as a valid form of adversarial attack. Following this line of research, FOA investigates this problem in VLMs, with a specific focus on degrading explanation faithfulness.

From this perspective, FOA is not merely a tool for probing correlations between saliency maps and model predictions. Rather, it actively optimizes input perturbations under a prediction-preservation constraint to induce explanation degradation. By incorporating our proposed optimization technique, FOA provides an efficient stress-testing framework for assessing the robustness and faithfulness of existing explanation methods under adversarial perturbations.

AuthorsOpenReview ↗

Weakness 1 and Weakness 4 - Evaluation Weakness and Adversarial Budget

Metric Weakness

The approach is based on directly optimizing for the Insertion-deletion metric, which is just one of the metrics and has it pros and cons. The success case is mostly made by presenting AUC for the metric that the approach is directly optimized for.

We thank the reviewers for raising this concern. FOA does not rely solely on the metric used during optimization to establish its effectiveness. In addition to the insertion-deletion-based objective, we evaluate the resulting explanations using independent faithfulness metrics, including DelmeanDel_{\text{mean}} and ROAD. Although both metrics follow a perturbation-based evaluation paradigm, they employ different imputation strategies: DelmeanDel_{\text{mean}} uses channel-wise mean imputation, whereas ROAD adopts Noisy Linear Imputation (NLI).

Besides, we also evaluate explanation shifts using the Spearman Rank Correlation (SRC), an additional metric that is not directly optimized by FOA. The observed changes in SRC further demonstrate FOA's effectiveness in altering the relative ranking of salient features beyond its optimization objective.

The consistent degradation observed across these complementary evaluations suggests that our findings are not merely an artifact of optimizing the insertion-deletion objective or a particular imputation strategy. This provides further evidence for the validity of our evaluation and supports the effectiveness of FOA in compromising explanation faithfulness.

Comparative Evaluation of FOA and X-Shift

Evaluations are not convincing. For benchmarking only x-shift is used. Whereas the authors note that this is currently the only method, in Tab 1, that method seems to have very poor results. Often distorting metric values worse than 'without attack'. Comparison against such a method does not make a convincing case, especially when AUC curves mainly compared with x-shift. Not to mention, the proposed approach is optimized against the insertion-deletion metric, hence its no surprise that its AUC curves are better.

We respectfully clarify that our evaluation is not limited to comparing FOA against X-Shift on faithfulness metrics. First, as discussed above, X-Shift primarily focuses on manipulating explanation maps, rather than directly evaluating the degradation of explanation faithfulness. Therefore, its relatively weak performance on faithfulness metrics should be interpreted in the context of its different optimization objective.

Second, as discussed in our response to the Metric Weakness above, to address the concern that FOA may achieve better AUC results simply because it is optimized against the insertion-deletion objective, we additionally evaluate FOA using independent faithfulness metrics, namely Del$_\text{mean}$ and ROAD. These metrics employ different perturbation-based evaluation strategies and are not directly optimized by FOA. Moreover, we also compare FOA against X-Shift in terms of explanation manipulation, which is the primary strength of X-Shift. Specifically, we evaluate the changes in saliency rankings using Spearman rank correlation (SRC). Our results show that FOA achieves stronger explanation manipulation than X-Shift, demonstrating that FOA is not only effective at degrading explanation faithfulness but also competitive in directly manipulating explanation maps.

Adversarial Budget

Also, 16/255 is a significant adversarial budget.

We acknowledge that an ℓ∞\ell_\infty perturbation budget of ϵ=16/255\epsilon=16/255 is relatively large. Nevertheless, this budget has also been adopted in prior adversarial attack studies (e.g., https://arxiv.org/abs/2507.07709, https://arxiv.org/abs/2204.00089). Moreover, under this evaluation setting, the resulting adversarial images remain visually reasonable, retaining their overall appearance and semantic content despite the applied perturbations. We therefore consider ϵ=16/255\epsilon=16/255 a reasonable setting for evaluating the effectiveness of FOA under a relatively strong perturbation budget.

To further address the concern regarding perturbation magnitude, we have conducted additional experiments with smaller budgets, ϵ∈[2/255,4/255,8/255]\epsilon \in [2/255, 4/255, 8/255], and report the results in Appendix E. While we use ϵ=16/255\epsilon=16/255 in our main experiments to demonstrate the attack's potential to substantially degrade explanation faithfulness, these additional experiments provide insights into its effectiveness under more restrictive perturbation budgets.

AuthorsOpenReview ↗

Weakness 2 and Weakness 3 - Clarification of Novelty and Explanation Methods for VLMs

Weakness 2

The claim that it is the first study to investigate the faithfulness of VLM explanations is objectionable in the presence of x-shift work.

Although X-Shift explicitly uses the term "explanation faithfulness" in its paper, as reflected in the statement that "explanation maps can be systematically manipulated while preserving the model’s original prediction, revealing a disconnect between predictive behavior and explanation faithfulness," (see https://arxiv.org/abs/2605.16651) both its discussion and experimental evaluation primarily focus on quantifying the extent of explanation-map shifts, rather than directly assessing whether the faithfulness of these explanations is actually degraded. An explanation shift does not necessarily imply reduced faithfulness, as demonstrated by our experiments and illustrated in Section 1. Introduction and Figure 1 as our motivation.

Therefore, X-Shift does not directly investigate the degradation of VLM explanation faithfulness, which is the central focus of our work. In this sense, we maintain that FOA is the first study to specifically investigate adversarial attacks on VLM explanation faithfulness.

Weakness 3

The writing is often misleading, e.g., the very first sentence referring to approaches appearing in 2018 and 2017 as VLM explanation methods is technically incorrect.

We thank the reviewer for pointing this out. We agree that our original sentence grouped explanation methods for conventional vision models and VLMs together, which could incorrectly imply that all cited methods were originally developed for VLMs. We have corrected this statement in the revised manuscript to distinguish the two categories explicitly:

'Recent explanation methods for Vision Models (Selvaraju et al., 2017; Chefer et al., 2021; Petsiuk et al., 2018) and Vision-Language Models (VLMs) (Zhao et al. (2024); Wang et al. (2023); Dong et al. (2023)) have demonstrated high explanation faithfulness by highlighting image regions that contribute to the predicted image-text similarity.'

Reviewer_JxgHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FOA, a prediction-preserving attack on VLM explanations. The main idea is to optimize insertion and deletion objectives while stopping gradients through saliency-map generation. Experiments with five explainers on CLIP ViT-B/16 across three datasets show substantial degradation in the evaluated faithfulness scores and transfer between explainers.

Critical strengths
  • The paper studies a meaningful limitation of existing explanation attacks that changing a saliency map does not necessarily make it less faithful.
  • The attack methods can handle both gradient-based and sampling-based explainers without requiring separate optimization.
  • Experimental results show stronger attack peformance compared with baseline X-Shift under the reported faithfulness metrics, with transfer across explainers.
Critical weaknesses
  • It is unclear whether the results mainly reflect misleading explanations or changes in the model’s response to masked images. FOA can directly optimize these responses, but controls with fixed clean or random saliency rankings are missing.
  • The independent faithfulness evaluation is not sufficiently convincing. The reported ROAD uses channel-wise mean filling, which differs from the debiased imputation in the original ROAD implementation, leaving the masking concern insufficiently addressed.
  • The comparison with X-Shift does not report changes in prediction probabilities. It remains unclear whether FOA’s advantage holds at comparable prediction stability.
Questions for authors
  • How much degradation remains when the clean saliency ranking is kept fixed on adversarial images? A random-ranking control would also help distinguish ranking changes from masked-input effects.
  • How would the ROAD results change with the original imputation method instead of mean filling?
  • Do the findings still hold on another backbones under the same perturbation and prediction-preservation constraints?
4 comments
AuthorsOpenReview ↗

Weakness 1 and Question 1 - Controlled Ablation on Saliency Ranking Strategies

Weakness 1

It is unclear whether the results mainly reflect misleading explanations or changes in the model’s response to masked images. FOA can directly optimize these responses, but controls with fixed clean or random saliency rankings are missing.

Question 1

How much degradation remains when the clean saliency ranking is kept fixed on adversarial images? A random-ranking control would also help distinguish ranking changes from masked-input effects.

We thank the reviewer for pointing out the need to disentangle the contribution of saliency ranking changes from changes in the model's responses to masked inputs. For the reviewer's recommendations, we conduct a controlled ablation study comparing three masking strategies during attack optimization (Appendix D, Table 4): (1) Adaptive-mask, which dynamically updates saliency rankings at each iteration; (2) Clean-mask, which fixes the ranking computed from the clean image throughout optimization; and (3) Random-mask, which uses a fixed random ranking (run in 3 independent seeds)

For ease of writing the response, we provide a reorganized and more compact presentation of the results from Table 4 below, with separate tables for the three masking strategies. The complete results are reported in Appendix D, Table 4.

The results reveal that both factors contribute to faithfulness degradation, with adaptive ranking providing an additional advantage. In particular, Adaptive-mask consistently achieves stronger faithfulness degradation than Clean-mask under ROAD (the new ROAD implementation – Noisy Linear Imputation; see our response to Weakness 2 and Question 2 (https://openreview.net/forum?id=sQeTssIaEL&noteId=hqu7VtTEn2)) across all three attack variants. This demonstrates that dynamically updating saliency rankings plays an important role in identifying and exploiting vulnerable image regions throughout optimization. Besides, Clean-mask still achieves non-negligible faithfulness degradation despite keeping the saliency ranking fixed. Since the masked regions remain unchanged while the adversarial image and the corresponding model responses evolve throughout optimization, this result indicates that changes in model responses to masked inputs also contribute to the degradation, independently of adaptive ranking updates. The weaker performance of Random-mask suggests that such degradation cannot be explained solely by changes in model responses to arbitrary masked regions; identifying informative regions through saliency-based rankings is also important.

These findings clarify that FOA's effectiveness arises from two complementary mechanisms: exploiting changes in model responses to masked inputs and adaptively updating saliency rankings to identify more informative regions during optimization.

Table 4(a). Ablation study of saliency ranking strategies under IOA. Results are obtained on CLIP VLMs using GradECLIP, averaged across three datasets and five explainers. Random-mask results are reported as mean ± standard deviation over three independent seeds.

Ranking StrategyIMD ↓SRC ↓Del_mean ↑ROAD ↑
Adaptive-mask0.25750.60880.12630.2241
Clean-mask0.27700.62120.11140.1502
Random-mask0.5702 ± 0.0090.8727 ± 0.0080.1178 ± 0.0060.1169 ± 0.005

Table 4(b). Ablation study of saliency ranking strategies under DOA. Results are obtained on CLIP VLMs using GradECLIP, averaged across three datasets and five explainers. Random-mask results are reported as mean ± standard deviation over three independent seeds.

Ranking StrategyIMD ↓SRC ↓Del_mean ↑ROAD ↑
Adaptive-mask0.29130.55080.33240.3303
Clean-mask0.60350.61730.18580.1771
Random-mask0.6328 ± 0.0030.7733 ± 0.0040.1654 ± 0.0010.1731 ± 0.001

Table 4(c). Ablation study of saliency ranking strategies under IOA+DOA. Results are obtained on CLIP VLMs using GradECLIP, averaged across three datasets and five explainers. Random-mask results are reported as mean ± standard deviation over three independent seeds.

Ranking StrategyIMD ↓SRC ↓Del_mean ↑ROAD ↑
Adaptive-mask0.43610.85000.14370.2521
Clean-mask0.44070.66430.19320.1755
Random-mask0.5937 ± 0.0200.8832 ± 0.0040.1080 ± 0.0000.1718 ± 0.005
AuthorsOpenReview ↗

Weakness 2 and Question 2 - ROAD Evaluation Protocol

Weakness 2.

The independent faithfulness evaluation is not sufficiently convincing. The reported ROAD uses channel-wise mean filling, which differs from the debiased imputation in the original ROAD implementation, leaving the masking concern insufficiently addressed.

Question 2.

How would the ROAD results change with the original imputation method instead of mean filling?

We thank the reviewer for identifying this issue in our previous ROAD implementation. We acknowledge that our previous implementation used channel-wise mean filling instead of the debiased imputation protocol adopted by the original ROAD method. To address this issue, we have corrected the evaluation and rerun all ROAD experiments using the original Noisy Linear Imputation (NLI) protocol. Specifically, the results previously reported as ROAD using channel-wise mean filling have been reassigned to Del$_{\text{mean}}$, while ROAD is now evaluated exclusively using the original NLI protocol.

In the revised manuscript, we have explicitly clarified the imputation protocols used by Del$_{\text{mean}}$ and ROAD in the Evaluation Metrics paragraph of Section 4.1. We have also updated the corresponding main-text and appendix tables, as well as the experimental analyses, to reflect this correction.

The corrected ROAD results continue to support our main findings, demonstrating that FOA remains effective in degrading explanation faithfulness. Furthermore, ROAD is excluded from FOA's optimization objective, providing an independent evaluation of faithfulness degradation under the original debiased imputation protocol.

AuthorsOpenReview ↗

Weakness 3 - Prediction Confidence

Weakness 3

The comparison with X-Shift does not report changes in prediction probabilities. It remains unclear whether FOA’s advantage holds at comparable prediction stability.

We thank the reviewer for highlighting the importance of prediction stability when comparing FOA with X-Shift. To address this concern, we additionally report the confidence of the original predicted class after each attack in Table 3 (Appendix C). For ease of comparison, we also provide Table 3 (Appendix C) below.

The results show that FOA achieves prediction confidence competitive with X-Shift and comparable to the clean baseline (Without Attack). This further establishes a fair comparison between the two methods and demonstrates that the observed faithfulness degradation is not merely a consequence of changes in model predictions.

Table 3. CLIP confidence scores of the original clean predicted class, averaged equally across selected explainers.

AttackImageNetStanfordPetsCUB
Without Attack0.98570.98430.8856
X-Shift0.99960.99930.9998
IOA0.95520.96970.8102
DOA0.99980.99890.9999
IOA+DOA0.74120.54410.5243
AuthorsOpenReview ↗

Question 3 - Attack Across Different VLM Backbones (BLIP and SigLIP)

Question 3

Do the findings still hold on another backbones under the same perturbation and prediction-preservation constraints?

We thank the reviewer for this suggestion. To examine whether FOA generalizes beyond the CLIP ViT-B/16 architecture, we have extended our experiments to two additional vision-language model (VLM) backbones under the same attack setting in Appendix F - Table 6: BLIP and SigLIP. Specifically, we evaluate both backbones across three datasets (ImageNet, StanfordPets, and CUB), considering five attack settings: Without Attack, X-Shift, IOA, DOA, and IOA+DOA. We assess their effectiveness using four faithfulness metrics: IMD, SRC, Del$_{\text{mean}}$, and ROAD. All results are averaged across explainers.

For ease of comparison, we provide a reorganized presentation of the results from Appendix F, Table 6, with separate tables for each VLM backbone (BLIP and SigLIP) and dataset (ImageNet, StanfordPets, and CUB).

As reported in Appendix F, Table 6, FOA consistently degrades explanation faithfulness across these additional backbones. These results provide further evidence that the vulnerabilities exposed by FOA are not specific to the CLIP architecture.

Table 6(a). Comparison of attack strategies on BLIP using ImageNet.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.2013--0.22800.2761
X-Shift0.16790.20250.22130.2915
IOA0.04040.19610.21180.2733
DOA-0.02750.23270.38470.4593
IOA+DOA-0.05470.22480.25160.2841

Table 6(b). Comparison of attack strategies on BLIP using StanfordPets.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.1642--0.27390.3073
X-Shift0.20020.21330.31700.3999
IOA0.04360.18940.23070.2874
DOA-0.02670.21400.51700.5806
IOA+DOA-0.10050.19710.31610.3515

Table 6(c). Comparison of attack strategies on BLIP using CUB.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.2414--0.20810.2483
X-Shift0.22780.24330.24700.2989
IOA0.11790.23070.24710.3063
DOA0.01660.27250.37790.4372
IOA+DOA-0.07020.22270.27490.3082

Table 6(d). Comparison of attack strategies on SigLIP using ImageNet.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.2847--0.28380.2594
X-Shift0.36460.14720.20960.2799
IOA0.00510.12430.22320.2778
DOA0.01100.14170.53250.5970
IOA+DOA-0.22300.13850.32330.3492

Table 6(e). Comparison of attack strategies on SigLIP using StanfordPets.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.3458--0.24560.2114
X-Shift0.35290.19720.20760.2701
IOA0.01210.17940.24660.2856
DOA-0.02470.10980.55070.6031
IOA+DOA-0.21310.16410.34110.3530

Table 6(f). Comparison of attack strategies on SigLIP using CUB.

AttackIMD ↓SRC ↓Del_mean ↑ROAD ↑
Without Attack0.3068--0.28840.2593
X-Shift0.40500.17480.22410.2906
IOA0.01750.14270.25260.3057
DOA0.03630.16480.56130.6199
IOA+DOA-0.23190.14130.35600.4009

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Responses to Reviewers

Dear ACs and Reviewers,

We have carefully revised our manuscript based on the reviewers' comments and suggestions. In addition to clarifying several aspects of our methodology and evaluation, we have conducted additional experiments and incorporated new analyses into the revised manuscript. We have also provided detailed responses to the questions and concerns raised by the reviewers.

For clarity, we summarize the major manuscript revisions and our responses to the reviewers' concerns below.

1. Revisions to the Manuscript

We have incorporated the following additional experiments and analyses into the revised manuscript:

  • Appendix C – Prediction Confidence: Added prediction confidence comparisons between FOA, X-Shift, and the clean baseline to further examine prediction preservation.

  • Appendix D – Controlled Ablation on Saliency Ranking: Added controlled experiments comparing Adaptive-mask, Clean-mask, and Random-mask strategies to disentangle the effects of adaptive saliency ranking and changes in model responses to masked inputs.

  • Appendix F – Generalization Across VLM Backbones: Extended experiments to two additional VLM backbones, BLIP and SigLIP, across three datasets to examine the generalizability of FOA beyond CLIP.

  • Appendix G – Hyperparameter Sensitivity Analysis: Added ablation studies on the step size α\alpha and number of optimization iterations KK to investigate the sensitivity of attack performance to these hyperparameters.

  • Evaluation Metrics Clarification: Clarified the distinction between Del$_{mean}$ and ROAD, corrected the ROAD implementation to follow the original Noisy Linear Imputation (NLI) protocol, and updated the corresponding experimental results and analyses.

2. Responses to Reviewers' Concerns

We summarize below how we addressed the concerns raised by each reviewer.

Reviewer JxgH (see https://openreview.net/forum?id=sQeTssIaEL&noteId=tupdvigUyb)

  • Sources of Faithfulness Degradation: Conducted controlled ablations with Adaptive-mask, Clean-mask, and Random-mask strategies to disentangle the effects of saliency ranking changes and masked-input model responses (Appendix D).

  • ROAD Evaluation: Corrected the ROAD implementation to follow the original Noisy Linear Imputation (NLI) protocol and updated the corresponding results and analyses.

  • Prediction Preservation: Added prediction confidence comparisons among the clean baseline, X-Shift, and FOA variants (Appendix C) to provide further evidence of prediction stability.

  • Generalization: Extended experiments to BLIP and SigLIP across three datasets, demonstrating that FOA's effectiveness is not limited to CLIP (Appendix F).

Reviewer 6Pgm (see https://openreview.net/forum?id=sQeTssIaEL&noteId=ATDFe4mMtg)

  • Motivation and Novelty: Clarified the adversarial threat posed by prediction-preserving explanation manipulation and FOA's role as a stress-testing framework. We also clarified its distinction from X-Shift, emphasizing that FOA explicitly targets faithfulness degradation.

  • Evaluation Metrics: Clarified the complementary roles of Del$_{mean}$, ROAD, and SRC, and emphasized that ROAD serves as an independent evaluation metric rather than part of FOA's optimization objective.

  • Perturbation Budget and Prediction Preservation: Clarified the rationale for the adopted perturbation budgets and discussed the existing results in Appendix E. We also explained how KL-divergence regularization and candidate selection help preserve the original predicted class.

  • Related Work: Revised the introduction to better distinguish explanation methods for conventional vision models from those specifically designed for VLMs.

Reviewer n3Hf (see https://openreview.net/forum?id=sQeTssIaEL&noteId=Yl945WWVDn)

  • Faithfulness Degradation and Generalization: Added controlled ablations to analyze the sources of faithfulness degradation and extended the evaluation to BLIP and SigLIP across three datasets (Appendices D and F).

  • Hyperparameter Sensitivity: Added analyses of the step size and optimization iterations (Appendix G), showing that attack effectiveness varies across configurations without consistent monotonic improvement.

  • Prediction Preservation: Clarified the candidate-selection mechanism in Algorithm 1 and provided additional prediction confidence analyses (Appendix C).

  • Surrogate Gradient: Clarified that the surrogate gradient is a deliberate design choice, not merely a computational-saving technique, enabling FOA to optimize faithfulness without requiring end-to-end differentiation through the explanation pipeline or modifying pretrained VLM architectures.

We sincerely thank the AC and all reviewers for their constructive feedback and valuable suggestions. Their comments have helped us improve the clarity, analysis, and empirical support of our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.