acceptodds
Under review as a conference paper at ICLR 2027

Attention Shrinkage for Vision Transformers via Posterior Weighting

Abstract

Vision Transformers (ViTs) utilize multi-head self-attention to capture global context, yet their raw attention scores are inherently noisy and densely distributed, often causing attention to bleed into irrelevant background elements and thereby limiting model interpretability. Regularization of attention scores is understudied, while existing methods require architectural modification, use annotated segmentation, or aggressively hard-thresholding. To address these limitations, we propose adaptive posterior weighting methods — specifically Posterior Probability Shrinkage (PPS) and Wiener Shrinkage — which act as robust, data-driven regularizers. By penalizing attention values proportionally to their likelihood of belonging to a null distribution or by attenuating noisy spectral components, our approaches probabilistically refine attention maps. To overcome the computational burden of resampling, we introduce a joint bootstrap distribution that aggregates null statistics across the dataset. While architecture-agnostic, our proposed posterior weighting improves sparsity, shrinkage, and interpretability. _Without optimizing for localization or segmentation_, regularized attention maps enhance spatial coherence and smoothness. Evaluated on DUTS (a subset of the ImageNet) and LIDC-IDRI (thoracic CT scans) using three ViT architectures, our automated calibration and posterior weighting significantly improve the localization of target regions. Overall, the proposed regularization techniques for attention scores offer a superior balance by producing substantially smoother, less noisy attention distributions, without any hyperparameter tuning. These benefits are pronounced in architectures with inherently noisier attention mechanisms, demonstrating a statistically principled and robust path toward more interpretable visual representations.

Primary area:
interpretability and explainable AI
Keywords:
Vision transformers, Attention, Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WyNcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a posterior-weighting-based attention regularization framework for Vision Transformers, aiming to suppress noisy attention responses and improve spatial concentration. The proposed framework consists of two main components. First, Posterior Probability Shrinkage (PPS) estimates the null distribution of attention scores through bootstrapping and applies continuous shrinkage using posterior weights derived from local false discovery rates. Second, Wiener Shrinkage estimates the noise power spectral density from bootstrapped attention maps and performs adaptive filtering in the frequency domain. The authors further introduce a Joint Bootstrap Distribution to reduce the computational cost of repeated resampling. Experiments are conducted on DUTS and LIDC-IDRI using Vanilla ViT, DeiT, and DINOv2. The results show improvements in attention concentration and smoothness in some settings, while the gains in target localization performance are inconsistent.

Critical strengths
  1. The paper proposes an attention post-processing framework that does not require additional training or modifications to the original model architecture. PPS and Wiener Shrinkage regularize attention scores from statistical probability and spatial frequency perspectives, respectively, offering potential practical value.
  2. The experiments cover both natural and medical image domains and evaluate three different ViT architectures using multiple metrics, providing a reasonably broad experimental evaluation. Joint Bootstrap also offers a practical approach to reducing computational overhead.
Critical weaknesses
  1. The construction of bootstrap-based null distributions for attention scores has already been explored by Chung and Laletin (2026), while empirical Bayes estimation and Wiener filtering are well-established statistical techniques. Therefore, the main contribution of this paper appears to be an extension and combination of existing methods. Furthermore, the paper lacks comparisons with simple denoising baselines, such as Gaussian smoothing and conventional frequency-domain filtering, making it difficult to establish the necessity of the proposed statistical shrinkage mechanisms.
  2. Appendix A.2.2 states that LIDC-IDRI uses nodule-level rather than scan-level splitting, meaning that different nodules from the same CT scan may appear in different data partitions. In addition, Appendix A.3 indicates that the models are fine-tuned using 683 images from fold 1, while the localization evaluation aggregates test samples from folds 1-4. If the same fine-tuned model weights are used across these evaluations, some evaluation samples may have already been included in training. The authors should clarify the relationship between model training and evaluation across folds and rule out potential data leakage. Otherwise, the reliability of the medical imaging results remains uncertain.
Questions for authors
  1. Appendix A.3 states that the models are fine-tuned using data from fold 1, while the localization evaluation includes test samples from folds 1-4. Could the authors clarify whether separate models are trained for each evaluation fold and confirm that there is no overlap between the training and evaluation sets?
  2. Both PPS and Wiener Shrinkage are based on established statistical techniques. Could the authors provide direct comparisons with Gaussian smoothing, conventional Wiener filtering, or other simple soft-thresholding methods to demonstrate the independent benefits of bootstrap-based posterior estimation?
2 comments
AuthorsOpenReview ↗

Denoising baselines and additional fidelity analysis

W1. Lack of denoising baselines

We thank the reviewer for this insightful comment. Indeed, our choice of baseline methods is quite specific and limited, hence, following the suggestions of the reviewers, we implemented additional baseline methods. We agree that Gaussian smoothing and a conventional frequency-domain filtering (which we refer to as base Wiener filtering for white noise) is an appropriate choice of baselines to compare our methods to. The Gaussian smoothing exploits a kernel of size 3x3 pixels with σ=1.0\sigma = 1.0, while the basic Wiener filtering estimates the white noise std from the image (the prior on noise power-spectral density (PSD) is gaussian). We apply these baseline methods to the attention maps of the images from the LIDC-IDRI test subset obtained with the ViT fine-tuned on the training subset of this dataset (please see more details about this renewed experiment below). In the table below we present the results for the mean values (and the corresponding std) of the ratios of the main metrics (ER, AGM and AUPRC; after regularization / before regularization) that we consider in our study along the results of our proposed methods.

Table 2: Metric ratios after / before regularization (mean ± std)

MethodER ratio ↑AGM ratio ↓AUPRC ratio ↑
Wiener (ours)1.124 ± 0.0230.774 ± 0.0261.004 ± 0.002
PPS (ours)1.124 ± 0.0560.942 ± 0.0051.000 ± 0.003
Gauss0.9983 ± 0.00010.9842 ± 0.00011.0011 ± 0.0001
Wiener (base)1.0000 ± 0.00011.0000 ± 0.00011.0000 ± 0.0001

Both of our proposed methods increase the ER by about 12% and reduce the AGM by 23% for Wiener and 6% for PPS. AUPRC stays essentially unchanged showing that these gains in concentration and smoothness come without any loss in how well the maps localize the nodule. In contrast, the standard baselines barely alter the maps: Gaussian smoothing lowers AGM by only 1.6% and slightly reduces the ER (0.998), while the base Wiener filter leaves all three metrics unchanged (1.000).

The fidelity results, reported in Table 1 in the reply to the reviewer TiHu above, further confirm that our proposed methods perform better than the baselines. In particular, we have measured how the attention scores (from attention maps or regularized attention maps) faithfully influence ViT predictions. Our faithfulness evaluation suggest that both of our proposed methods improve the fidelity of the attention maps. Our Wiener filter on the bootstrap yields the largest gain: it raises the median fidelity from 0.348 to 0.408 (median per-image difference +0.099), achieves higher fidelity than the original map on 57.4% of images (rank-biserial r = +0.21), and the improvement remains significant after Holm correction (p = 0.0007). PPS also improves fidelity significantly (median difference +0.110, higher fidelity on 55.9% of images, r = +0.15, p_Holm = 0.026). In contrast, the two standard denoising baselines, Gaussian smoothing and the white-noise Wiener filter, significantly reduce fidelity (r = −0.25 and −0.23, p_Holm < 0.0001 and 0.0002), yielding higher fidelity than the original map on only 40.5% and 41.8% of images. Generic smoothing therefore makes the attention maps less faithful to the model, whereas the proposed statistically grounded methods make them more faithful.

Taken together, these results highlight the necessity of the proposed statistical shrinkage mechanism – we believe that the probabilistic nature of these methods allow them to noticeably outperform the new baseline methods. In the revised manuscript, we will describe these baselines in detail and report their results across all models and datasets considered in our study.

W2. Potential data leakage

We thank the reviewer for this insightful comment. Indeed, in our original experiment on LIDC-IDRI the splitting was performed at the nodule level and, furthermore, we aggregated several test folds to construct the evaluation dataset for our regularization methods. It is possible that a data leakage could have occurred in our original study, although we mainly focused on the characteristics of the original and regularized attention maps and the classification accuracy of the model was not essential for our goals.

Nevertheless, we corrected the construction of the LIDC-IDRI to exclude a potential data leakage and the additional experiments that we describe in our replies were conducted on the dataset that employs patient-level splitting. The evaluation dataset now contains only one test fold with 170 nodules from 106 patients, which are sliced into 510 images (central slices of each view, 246 malignant, 264 benign), out of which 467 images with annotation masks larger than 490 pixels were selected. The training fold, which is used to fine-tune a model, consists of 684 nodules from 417 patients and the 2D slices are chosen uniformly at random from 3 views and distances from the nodule center at each training epoch. We will retrain all our models on this new realization of the LIDC-IDRI dataset and reproduce all our experiments with it. The detailed description of the new data partitioning, as well as the results obtained on it, will be included in the revised version of our manuscript.

Q1, Q2

We believe that we have answered these questions in our replies to the critical weaknesses pinpointed by the reviewer. However, if any questions have not been fully addressed by us, we kindly ask the reviewer to indicate that in the comment to our reply.

Reviewer_p6szOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the problem of noisy and densely distributed attention maps in ViTs, which limits their interpretability. The authors propose two adaptive posterior weighting methods: Posterior Probability Shrinkage(PPS) and Wiener Shrinkage. PPS uses a bootstrap-estimated null distribution to compute posterior probabilities and adaptively suppress attention scores, while Wiener Shrinkage suppresses noise in the frequency domain. To reduce the computational cost of resampling, this work also introduces a joint bootstrap distribution that aggregates statistics across the dataset. Experiments are conducted on natural images and medical images using three classical architectures: ViT, DeiT, and DINOv2. The results show that PPS and Wiener Shrinkage effectively produce more concentrated, sparse, and smooth attention maps.

Critical strengths
  1. The proposed methods are straightforward and can be easily applied to different ViT architectures.
  2. The experiments cover multiple model architectures and datasets from different domains, with a diverse set of evaluation metrics.
Critical weaknesses
  1. Although the paper provides an extensive analysis of the statistical properties of attention maps, the connection between these properties and model interpretability remains insufficiently established and requires further clarification.
  2. The baseline comparisons are relatively limited, focusing primarily on hard-thresholding methods without considering other soft-shrinkage baselines. It remains unclear whether the proposed statistical weighting methods offer significant advantages over simpler continuous regularization approaches.
Questions for authors
  1. How do improvements in attention map sparsity and smoothness contribute to better interpretability?
  2. How do the proposed methods compare with simpler soft-shrinkage or smoothing baselines?
6 comments

W1. Lack of connection between the presented results and model interpretability

We thank the reviewer for this comment. We agree that a demonstration of how the proposed methods impact a model’s performance should fortify our claims about the improved interpretability. To directly assess this, we compute the fidelity of the regularized attention maps, which quantify how image features according to an attention map are influencing the model classification [1]. Image features are perturbed one by one in the order given by their attention weights, first from lowest to highest and then from highest to lowest, while the model’s logits for the target class are tracked. Fidelity is defined as the area between the two resulting logit curves. We compute it for both the original and the regularized attention maps and apply a paired test across the dataset to determine whether regularization significantly improves fidelity. Below are the results of these comparison for fine-tuned ViT on LIDC-IDRI dataset which suggest that the proposed methods systematically improve fidelity w.r.t. the original attention maps and the baseline methods that we consider.

Table 1: Fidelity: Wilcoxon signed-rank test vs. the original attention map (am), Holm-corrected, α = 0.05

Methodnmedian (am)median (method)median Δmean Δ% higherrank-biserialpp (Holm)result
Wiener4670.34830.40810.09900.101157.4%+0.209<0.00010.0007↑ higher
z > 04670.34830.38330.05380.084753.5%+0.1540.00380.0262↑ higher
PPS4670.34830.38300.11030.085655.9%+0.1510.00470.0262↑ higher
PPS + π₀4670.34830.37640.07570.077355.0%+0.1300.01520.0608n.s.
π₀4670.34830.37640.07570.077155.0%+0.1290.01550.0608n.s.
Wiener (base)4670.34830.3358−0.0013−0.002841.8%−0.229<0.00010.0002↓ lower
Gauss4670.34830.3200−0.0207−0.018940.5%−0.250<0.0001<0.0001↓ lower

For more details regarding this analysis please see the beginning of our reply to reviewer TiHu. The description of this analysis as well as the results for all the models and datasets that we consider will be included in the revised version of the manuscript.

[1] Brocki, Chung (2023) Fidelity of Interpretability Methods and Perturbation Artifacts in Neural Networks. Tiny Paper at ICLR 2023

W2. Limited baseline comparisons

We thank the reviewer for this insightful comment. Accordingly, we decided to implement additional baseline methods. As baselines we have adapted Gaussian smoothing (3×3 kernel, σ = 1.0) and a standard Wiener filter for white noise. The noise standard deviation is estimated from the image, and the PSD prior is a simple Gaussian rather than one built from bootstrap images, as in our proposed Wiener shrinkage method. Both baselines are applied to the attention maps of the LIDC-IDRI test images, obtained from a ViT fine-tuned on the training subset (see our reply to reviewer WyNc for details of this updated experiment).

Metric ratios after / before regularization (mean ± std)

MethodER ratio ↑AGM ratio ↓AUPRC ratio ↑
Wiener1.124 ± 0.0230.774 ± 0.0261.004 ± 0.002
PPS1.124 ± 0.0560.942 ± 0.0051.000 ± 0.003
Gauss0.9983 ± 0.00010.9842 ± 0.00011.0011 ± 0.0001
Wiener (base)1.0000 ± 0.00011.0000 ± 0.00011.0000 ± 0.0001

Table 2 reports how each method changes the attention maps relative to the original map (after / before; 1 means no change). Both proposed methods increase the energy ratio by about 12% (1.124 for Wiener and PPS), and reduce the average gradient magnitude, by 23% for Wiener (0.774) and 6% for PPS (0.942). AUPRC stays essentially unchanged (1.004 and 1.000), showing that these gains in concentration and smoothness come without any loss in how well the maps localize the nodule. In contrast, the standard baselines barely alter the maps: Gaussian smoothing lowers AGM by only 1.6% and slightly reduces the energy ratio (0.998), while the white-noise Wiener filter leaves all three metrics unchanged (1.000).

The results of the fidelity analysis that further support a better performance of our proposed methods w.r.t. the baseline methods can be found in Table 1 in our reply above. Altogether, these results demonstrate that our regularization methods outperform the new baselines by a wide margin.

The proper description of these new baseline methods together with the results of their application to all the models and datasets considered in our study (that we are currently working on) will be included in the revised version of our manuscript.

Q1. How do improvements in attention map sparsity and smoothness contribute to better interpretability?

In machine learning, sparsity has long been a foundational assumption of simplicity that isolates the most important predictors, making the underlying signal easier to interpret [1]. In practically all cases, an attention map of ViT consists of non-zero attention scores, implying all image features contribute to the classification. Sparsity and smoothness address two common shortcomings of raw attention maps. Sparsity concentrates attention on a smaller number of features, so it is easier to see which regions the model relies on. Smoothness suppresses spurious, high-frequency variations between neighboring features, so that the maps highlight coherent regions rather than isolated, noisy peaks. Together, these properties make the maps easier to read and less ambiguous to inspect. We do not claim that this alone necessarily makes the model more interpretable, although we consider it an essential step towards a better interpretability. To strengthen our claims, we complement the visual and statistical analysis with a fidelity evaluation (please see our reply to the critical weaknesses above), which tests whether the regularized maps still reflect the features that drive the model’s predictions. In the revised manuscript, we will state this more carefully and present sparsity and smoothness as properties that improve the readability of attention maps, rather than as direct evidence of improved interpretability.

[1] Hastie, Tibshirani, Wainwright (2015) Statistical Learning with Sparsity

Q2. How do the proposed methods compare with simpler soft-shrinkage or smoothing baselines?

We believe that we address this question fully in our reply to the critical weaknesses above. To summarize it here, the results of our supplementary study on LIDC-IDRI with Vanilla ViT demonstrate that our proposed methods substantially outperform both additional baselines in terms of ER and AGM metrics and the improved interpretability claim is corroborated by our fidelity analysis.

Reviewer_p6szOpenReview ↗

Response to Author Rebuttal

Thanks for the additional experiments and clarifications. The fidelity evaluation adequately addresses my first concern. W2 is only partially addressed, as the new baselines barely modify the attention maps and include no soft-shrinkage baseline.

I appreciate the authors' efforts and will maintain my original score.

Reviewer_TiHuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes two post-hoc methods for regularizing Vision Transformer (ViT) attention maps: Posterior Probability Shrinkage (PPS) and Wiener Shrinkage. Both approaches use bootstrap attention distributions generated from pixel-shuffled input images to estimate a statistical noise reference. PPS attenuates attention coefficients according to estimated posterior probabilities, while Wiener Shrinkage suppresses noise-associated components in the Fourier domain. The authors additionally propose a joint bootstrap strategy to reduce computational overhead.

The methods are evaluated on natural images (DUTS) and medical CT images (LIDC-IDRI), using ViT, DeiT, and DINOv2. Evaluation focuses on attention sparsity, concentration, spatial smoothness, and localization relative to annotated object regions. The authors report improvements in attention concentration and, in some settings, object localization, although improvements vary across models, datasets, and evaluation metrics. The proposed methods operate on extracted attention maps without modifying the underlying model predictions.

Critical strengths

The methodology is reasonably coherent. Bootstrap-based null estimation, posterior probability shrinkage, and Wiener filtering form a mathematically understandable approach to post-hoc attention regularization. The authors demonstrate that these techniques change attention concentration, sparsity, and smoothness as intended.

Critical weaknesses
  1. Fundamental weakness: insufficient evidence that attention regularization improves model interpretability.

The main concern is that the manuscript does not convincingly establish that the proposed attention regularization methods improve the interpretability of ViT predictions. While the methods produce sparser, more concentrated, and sometimes smoother attention maps, it remains unclear whether these changes make the maps more informative or faithful to the model's underlying decision process.

First, the statistical assumption underlying the proposed methods is insufficiently justified. The authors construct a bootstrap null distribution by randomly shuffling input pixels, thereby destroying the spatial organization of the images. However, it is unclear whether the resulting attention distribution provides a meaningful representation of task-irrelevant attention in pretrained or fine-tuned ViTs. This is particularly concerning for medical images, where anatomical structures and their spatial relationships carry important diagnostic information. Attention generated from spatially randomized, out-of-distribution images does not necessarily represent irrelevant attention in real images.

Second, the quantitative evaluation does not convincingly demonstrate improved interpretability. Figure 3 primarily relies on the energy ratio (ER), which measures the proportion of attention mass within annotated object regions. However, ER can increase simply because background attention is suppressed more strongly than foreground attention, even when the relative ranking of foreground and background pixels remains largely unchanged. For example, if an original attention map assigns 40% of its attention mass to the object, suppressing half of the background attention while preserving the foreground increases ER from 0.40 to approximately 0.57, a 43% relative improvement, without necessarily identifying more meaningful image features. Indeed, the supplementary AUROC/AUPRC results do not consistently support improved foreground discrimination. For DINOv2 on DUTS, PPS increases ER but reduces median AUROC from 0.961 to 0.943 and normalized AUPRC from 5.069 to 4.934.

Most importantly, improved localization within an annotated object does not necessarily imply improved explanation of the model's prediction. For example, in the LIDC-IDRI experiments, concentrating attention within a lung nodule does not establish which image characteristics contribute to the benign-versus-malignant classification. The manuscript does not demonstrate that the retained attention corresponds more closely to image information relevant to the model's decisions.

Overall, the experiments characterize how the proposed methods modify attention distributions, but they do not convincingly establish the practical or scientific benefit of these modifications for model interpretability. Consequently, the evidence supports a narrower contribution in post-hoc attention-map regularization than the broader interpretability claims made in the manuscript.

  1. The presentation weakness: The manuscript does not establish a sufficiently clear connection between the proposed methodological developments and their practical value for model interpretability. In particular, the presentation of qualitative results is surprisingly limited for a paper focused on improving visual attention maps. Among the four main figures, only Figure 1 provides a qualitative comparison of original and regularized attention maps, using a single natural-image example. No qualitative results on medical images are presented in the main figures, despite LIDC-IDRI being one of the two evaluation datasets. The medical-image visualizations are relegated to the supplementary material. The remaining main figures primarily report statistical properties of the regularized attention maps, such as sparsity, concentration, smoothness, and localization within annotated regions. Although these measurements characterize the effects of the proposed methods, they do not clearly demonstrate how the resulting attention maps provide more meaningful or useful interpretations of model predictions.
Questions for authors

I have no specific clarification questions at this stage. My main concerns relate to the scientific motivation, the evidence supporting the claimed interpretability improvements, and the overall significance of the contribution, as detailed in my review.

3 comments
AuthorsOpenReview ↗

Fidelity (faithfulness) evaluation as evidence that attention regularization improves model interpretability

We thank the reviewer for a very thoughtful and detailed review and insightful suggestions. We address the reviewer’s concerns point by point below.

W1. Insufficient evidence that regularization improves interpretability

We thank the reviewer for the suggestion. Accordingly, we perform a fidelity evaluation that aims to measure how the attention scores faithfully influence ViT predictions [1]. In brief, we obtain the model’s logits for the target class, while increasingly perturbing the image features in the most important first (MIF) and in the least important first (LIF) order. The fidelity metric is calculated as an area between the MIF and LIF curves. We calculated fidelity for the original and regularized attention maps, and applied a paired test across the dataset to assess whether regularization improves it.

In the table below, we present the results of a paired test for a vanilla ViT fine-tuned on the LIDC dataset (reconstructed to separate training and test folds by patients to prevent data leakage; for more details please see our reply to reviewer WyNc). Additionally, we compare our methods to two baseline denoising methods (that do not rely on bootstrapping): Gaussian smoothing and basic Wiener filtering for white noise (for more details please see our replies to reviewers p6sz and WyNC). Table 1 shows, for each regularization method, a paired comparison of its fidelity with that of the original attention map (am) on the same 467 test images, using a two-sided Wilcoxon signed-rank test with Holm correction for multiple comparisons (α = 0.05). The columns give the median fidelity of am and of the method, the median and mean per-image difference (method - am), the share of images on which the method achieves higher fidelity, the matched-pairs rank-biserial effect size (from -1 to 1, positive when the method scores higher), the raw and Holm-adjusted p-values, and the resulting verdict (significantly higher, significantly lower, or not significant).

Table1: Fidelity: Wilcoxon signed-rank test vs. the original attention map (am), Holm-corrected, α = 0.05

Methodnmedian (am)median (method)median Δmean Δ% higherrank-biserialpp (Holm)result
Wiener4670.34830.40810.09900.101157.4%+0.209<0.00010.0007↑ higher
z > 04670.34830.38330.05380.084753.5%+0.1540.00380.0262↑ higher
PPS4670.34830.38300.11030.085655.9%+0.1510.00470.0262↑ higher
PPS + π₀4670.34830.37640.07570.077355.0%+0.1300.01520.0608n.s.
π₀4670.34830.37640.07570.077155.0%+0.1290.01550.0608n.s.
Wiener (base)4670.34830.3358−0.0013−0.002841.8%−0.229<0.00010.0002↓ lower
Gauss4670.34830.3200−0.0207−0.018940.5%−0.250<0.0001<0.0001↓ lower

The results in Table 1 suggest that both proposed methods improve the fidelity of the attention maps. Our Wiener filter (based on the bootstrap noise estimate) yields the largest gain: it raises the median fidelity from 0.348 to 0.408 (median per-image difference +0.099), achieves higher fidelity than the original map on 57.4% of images (rank-biserial r = +0.21), and the improvement remains significant after Holm correction (p = 0.0007). PPS also improves fidelity significantly (median difference +0.110, higher fidelity on 55.9% of images, r = +0.15, p_Holm = 0.026), as does the z > 0 regularization on which it builds. In contrast, the two standard denoising baselines, Gaussian smoothing and the white-noise Wiener filter, significantly reduce fidelity (r = −0.25 and −0.23, p_Holm < 0.0001 and 0.0002), yielding higher fidelity than the original map on only 40.5% and 41.8% of images. Generic smoothing therefore makes the attention maps less faithful to the model, whereas the proposed statistically grounded methods make them more faithful.

We are performing the same fidelity analysis for all the models and datasets that we use in our study and we will report the results alongside a detailed description of these experiments in the revised version of our manuscript.

[1] Brocki, Chung (2023) Fidelity of Interpretability Methods and Perturbation Artifacts in Neural Networks. Tiny Paper at ICLR 2023

W1.1. Unjustified statistical assumption behind the proposed methods

We appreciate the reviewer’s comment. While the resulting null distribution might not be the most meaningful representation of the task-irrelevant attention, we would like to note that a) the attention generated from spatially randomized out-of-distribution images does reflect the model’s attention noise to a substantial degree by construction; b) our regularization procedures with these null distributions systematically improve the sparsity and concentration of the attention scores better than the baselines, which is further corroborated by the fidelity tests that we describe above (Table 1).

We agree that a targeted approach (e.g., for medical images with anatomical structures) could be developed and it may improve interpretability. In future work, we will consider alternative bootstrapping procedures including patch bootstrapping corresponding to the grid structure of ViT, bootstrapping residuals with estimated anatomical structures, and combination of the bootstrap with Monte Carlo simulation for human anatomy.

W1.2 Inconvincing quantitative evaluation

We thank the reviewer for this comment and observation. We agree that the improved interpretability claim should not rely on the ER enhancement alone and other metrics like AUPRC should be evaluated too. Nonetheless, we kindly suggest that in the given example being able to automatically "suppress half of the background attention while preserving the foreground attention" implies that the method improves the interpretability of the map in the sense that the irrelevant pixels (e.g. background) are discriminated from the relevant ones (e.g. foreground) and suppressed. This is an intended behavior of the ER metric. The additional study that we perform (as described above) demonstrates that:

  1. the baseline methods do not improve the ER at all (please see Table 2 in our reply to reviewer p6sz ) – and this aligns with a poorer fidelity;

  2. even when the relative ranking of foreground and background remain largely unchanged (as can be seen in Table 1 for the case of z-thresholding regularization) the fidelity improves.

We will add a more detailed description of the results of our quantitative evaluation including the limitations of the ER metric and the importance of other metrics in the revised version of our manuscript.

Most importantly, improved localization within an annotated object does not necessarily imply improved explanation of the model's prediction...

We agree that the demonstration of an improved model’s performance with the use of regularized attention maps would solidify our claims about interpretability. For that reason, we conducted the fidelity analysis described above. The results of this analysis show that our regularized attention maps are better at establishing which image features contribute to the benign-vs-malignant classification. When removing image features according to a regularized attention map (e.g., in the order of the most important feature first), a greater decrease in the model output (logits) shows that information relevant to the model’s decision was removed. We will include these results in the revised version of the manuscript.

Overall, the experiments characterize how the proposed methods modify attention distributions, but they do not convincingly establish the practical or scientific benefit of these modifications for model interpretability...

We thank the reviewer for this insightful comment. We will revise the text so that our claims reflect the specific contribution of this work. We have taken a data-driven probabilistic approach to regularize attention maps, whereas most of previous works in this area have modified ViT architecture or use segmentation. Please note that regularization in ML has long developed with interpretability in mind [1]. While our contribution is methodological in nature, comprehensive performance across sparsity, concentration, smoothness, and fidelity would provide an immediate use case for anyone using attention maps of ViTs. Upon acceptance, we plan to release our code repository to facilitate practical benefits of our regularization methods.

[1] Hastie, Tibshirani, Wainwright (2015) Statistical Learning with Sparsity

W2. The presentation weakness

We thank the reviewer for this comment. We agree that more figures in the main text would improve the quality of our explanations and better highlight our contribution. However, we could hardly include any more figures due to the length limit, hence we had to put them in the supplemental materials (D.1.1 DUTS and D.1.2 LIDC-IDRI). We will provide more qualitative examples including LIDC-IDRI considering that the camera-ready version will allow one additional page.

We also believe that the results of our fidelity analysis (described in our replies above) would establish a better connection between our methodological developments and their practical value for model interpretability, hence we will include them in the main text. In short, regularized attention maps provide more meaningful interpretations of model predictions, as our method is able to more accurately identify features that are important for tumor classification. We will revise the manuscript text to better align our results with the claims that we make.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.