- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the placement of local perturbations affects image classifiers under matched masking budgets. It compares attribution-guided and random masking across datasets, architectures, patch sizes, and perturbation operators, using confidence degradation curves and complementary summary metrics. The results show that no strategy is uniformly strongest: object-aware random masking can match or exceed attribution-based guidance in some settings, while SHAP induces stronger degradation in others.
Critical strengths
The paper compares attribution-guided and random masking across multiple datasets, architectures, and perturbation settings. Including object-aware random baselines provides a useful reference for assessing the added value of attribution-based guidance.
Critical weaknesses
-
Inconsistency between the metric definition and its interpretation. In Eq. (7), the point of the steepest confidence drop is defined using . However, since represents confidence, the steepest drop should correspond to the minimum derivative rather than the maximum. Moreover, the paper seems to mix up the location and magnitude of the confidence drop. It is unclear whether this is a typo or an implementation issue.
-
Numerical inconsistencies across tables. In Tables 2/9, the relative AUC for ResNet50 with MDP-32 and Random Object First is 0.00077, while Table 13 reports 0.0077 for the same setting, a tenfold difference. A similar issue appears for ViT. Additionally, some and values in Table 8 appear to be swapped compared with the main results. These inconsistencies raise concerns about the reliability of the reported results.
-
Questionable interpretation of the metrics. The paper generally assumes that lower indicates stronger degradation. However, this metric measures the variation across samples, and a lower value does not necessarily imply stronger degradation. For example, it could indicate either consistently low or consistently high confidence across samples. This interpretation needs further clarification, especially since the proposed diagnostic metrics are a key contribution of the paper.
Questions for authors
Seen above, this paper requires substantial rewriting, along with more rigorous and carefully conducted experiments.