Where to Place Local Perturbations? Breaking Image Classifiers

The placement of local disturbances is a critical factor in robustness testing for vision models. A straightforward strategy is random placement, while attribution methods can guide perturbations toward regions considered relevant for the model prediction. We study which placement strategies induce the strongest degradation under a fixed perturbation budget. Focusing on image classification as a controlled first setting, we compare attribution-based guidance with matched random baselines across datasets, architectures, perturbation operators, and spatial resolutions. We find that no guidance strategy is uniformly strongest. Object-aware random masking can approach or exceed attribution-based guidance in several ImageNet-S50 settings, whereas SHAP induces stronger degradation in other configurations, particularly on Oxford Flowers. Our results show that effective perturbation placement is strongly configuration-dependent and provide a basis for future robustness testing of more complex vision tasks.