AMEND: Attention Misalignment for Natural Adversarial Attacks against Object Detectors

Object detectors are increasingly deployed in safety-critical applications such as autonomous driving, surveillance, and robotics. However, their vulnerability to adversarial examples raises serious security concerns. Existing attacks face a fundamental trade-off between attack effectiveness and visual naturalness. Most methods optimize perturbations in image or pixel space, where semantic manipulation is inherently coupled with structural changes. Consequently, stronger attacks often introduce visible artifacts or unwanted distortions. In this paper, we argue that optimizing semantic representations rather than pixels provides a more controllable attack space. We formulate natural adversarial attacks as semantic alignment manipulation between generative models and discriminative detectors. Our key insight is that diffusion cross-attention enables semantic manipulation while preserving image structure. Based on this insight, we propose AMEND, a diffusion-based adversarial attack framework that integrates detector-derived conditioning, semantic attention dispersion, and structural consistency constraints. Experiments on seven object detectors show that AMEND achieves the lowest mAP on six of them, reducing relative mAP to as low as 6.73%, while consistently achieving superior visual naturalness across the evaluated settings.