- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a posterior-weighting-based attention regularization framework for Vision Transformers, aiming to suppress noisy attention responses and improve spatial concentration. The proposed framework consists of two main components. First, Posterior Probability Shrinkage (PPS) estimates the null distribution of attention scores through bootstrapping and applies continuous shrinkage using posterior weights derived from local false discovery rates. Second, Wiener Shrinkage estimates the noise power spectral density from bootstrapped attention maps and performs adaptive filtering in the frequency domain. The authors further introduce a Joint Bootstrap Distribution to reduce the computational cost of repeated resampling. Experiments are conducted on DUTS and LIDC-IDRI using Vanilla ViT, DeiT, and DINOv2. The results show improvements in attention concentration and smoothness in some settings, while the gains in target localization performance are inconsistent.
Critical strengths
- The paper proposes an attention post-processing framework that does not require additional training or modifications to the original model architecture. PPS and Wiener Shrinkage regularize attention scores from statistical probability and spatial frequency perspectives, respectively, offering potential practical value.
- The experiments cover both natural and medical image domains and evaluate three different ViT architectures using multiple metrics, providing a reasonably broad experimental evaluation. Joint Bootstrap also offers a practical approach to reducing computational overhead.
Critical weaknesses
- The construction of bootstrap-based null distributions for attention scores has already been explored by Chung and Laletin (2026), while empirical Bayes estimation and Wiener filtering are well-established statistical techniques. Therefore, the main contribution of this paper appears to be an extension and combination of existing methods. Furthermore, the paper lacks comparisons with simple denoising baselines, such as Gaussian smoothing and conventional frequency-domain filtering, making it difficult to establish the necessity of the proposed statistical shrinkage mechanisms.
- Appendix A.2.2 states that LIDC-IDRI uses nodule-level rather than scan-level splitting, meaning that different nodules from the same CT scan may appear in different data partitions. In addition, Appendix A.3 indicates that the models are fine-tuned using 683 images from fold 1, while the localization evaluation aggregates test samples from folds 1-4. If the same fine-tuned model weights are used across these evaluations, some evaluation samples may have already been included in training. The authors should clarify the relationship between model training and evaluation across folds and rule out potential data leakage. Otherwise, the reliability of the medical imaging results remains uncertain.
Questions for authors
- Appendix A.3 states that the models are fine-tuned using data from fold 1, while the localization evaluation includes test samples from folds 1-4. Could the authors clarify whether separate models are trained for each evaluation fold and confirm that there is no overlap between the training and evaluation sets?
- Both PPS and Wiener Shrinkage are based on established statistical techniques. Could the authors provide direct comparisons with Gaussian smoothing, conventional Wiener filtering, or other simple soft-thresholding methods to demonstrate the independent benefits of bootstrap-based posterior estimation?
2 comments
W2. Potential data leakage
We thank the reviewer for this insightful comment. Indeed, in our original experiment on LIDC-IDRI the splitting was performed at the nodule level and, furthermore, we aggregated several test folds to construct the evaluation dataset for our regularization methods. It is possible that a data leakage could have occurred in our original study, although we mainly focused on the characteristics of the original and regularized attention maps and the classification accuracy of the model was not essential for our goals.
Nevertheless, we corrected the construction of the LIDC-IDRI to exclude a potential data leakage and the additional experiments that we describe in our replies were conducted on the dataset that employs patient-level splitting. The evaluation dataset now contains only one test fold with 170 nodules from 106 patients, which are sliced into 510 images (central slices of each view, 246 malignant, 264 benign), out of which 467 images with annotation masks larger than 490 pixels were selected. The training fold, which is used to fine-tune a model, consists of 684 nodules from 417 patients and the 2D slices are chosen uniformly at random from 3 views and distances from the nodule center at each training epoch. We will retrain all our models on this new realization of the LIDC-IDRI dataset and reproduce all our experiments with it. The detailed description of the new data partitioning, as well as the results obtained on it, will be included in the revised version of our manuscript.
Q1, Q2
We believe that we have answered these questions in our replies to the critical weaknesses pinpointed by the reviewer. However, if any questions have not been fully addressed by us, we kindly ask the reviewer to indicate that in the comment to our reply.
Denoising baselines and additional fidelity analysis
W1. Lack of denoising baselines
We thank the reviewer for this insightful comment. Indeed, our choice of baseline methods is quite specific and limited, hence, following the suggestions of the reviewers, we implemented additional baseline methods. We agree that Gaussian smoothing and a conventional frequency-domain filtering (which we refer to as base Wiener filtering for white noise) is an appropriate choice of baselines to compare our methods to. The Gaussian smoothing exploits a kernel of size 3x3 pixels with σ=1.0, while the basic Wiener filtering estimates the white noise std from the image (the prior on noise power-spectral density (PSD) is gaussian). We apply these baseline methods to the attention maps of the images from the LIDC-IDRI test subset obtained with the ViT fine-tuned on the training subset of this dataset (please see more details about this renewed experiment below). In the table below we present the results for the mean values (and the corresponding std) of the ratios of the main metrics (ER, AGM and AUPRC; after regularization / before regularization) that we consider in our study along the results of our proposed methods.
Table 2: Metric ratios after / before regularization (mean ± std)
Both of our proposed methods increase the ER by about 12% and reduce the AGM by 23% for Wiener and 6% for PPS. AUPRC stays essentially unchanged showing that these gains in concentration and smoothness come without any loss in how well the maps localize the nodule. In contrast, the standard baselines barely alter the maps: Gaussian smoothing lowers AGM by only 1.6% and slightly reduces the ER (0.998), while the base Wiener filter leaves all three metrics unchanged (1.000).
The fidelity results, reported in Table 1 in the reply to the reviewer TiHu above, further confirm that our proposed methods perform better than the baselines. In particular, we have measured how the attention scores (from attention maps or regularized attention maps) faithfully influence ViT predictions. Our faithfulness evaluation suggest that both of our proposed methods improve the fidelity of the attention maps. Our Wiener filter on the bootstrap yields the largest gain: it raises the median fidelity from 0.348 to 0.408 (median per-image difference +0.099), achieves higher fidelity than the original map on 57.4% of images (rank-biserial r = +0.21), and the improvement remains significant after Holm correction (p = 0.0007). PPS also improves fidelity significantly (median difference +0.110, higher fidelity on 55.9% of images, r = +0.15, p_Holm = 0.026). In contrast, the two standard denoising baselines, Gaussian smoothing and the white-noise Wiener filter, significantly reduce fidelity (r = −0.25 and −0.23, p_Holm < 0.0001 and 0.0002), yielding higher fidelity than the original map on only 40.5% and 41.8% of images. Generic smoothing therefore makes the attention maps less faithful to the model, whereas the proposed statistically grounded methods make them more faithful.
Taken together, these results highlight the necessity of the proposed statistical shrinkage mechanism – we believe that the probabilistic nature of these methods allow them to noticeably outperform the new baseline methods. In the revised manuscript, we will describe these baselines in detail and report their results across all models and datasets considered in our study.