acceptodds
Under review as a conference paper at ICLR 2027

AMEND: Attention Misalignment for Natural Adversarial Attacks against Object Detectors

Abstract

Object detectors are increasingly deployed in safety-critical applications such as autonomous driving, surveillance, and robotics. However, their vulnerability to adversarial examples raises serious security concerns. Existing attacks face a fundamental trade-off between attack effectiveness and visual naturalness. Most methods optimize perturbations in image or pixel space, where semantic manipulation is inherently coupled with structural changes. Consequently, stronger attacks often introduce visible artifacts or unwanted distortions. In this paper, we argue that optimizing semantic representations rather than pixels provides a more controllable attack space. We formulate natural adversarial attacks as semantic alignment manipulation between generative models and discriminative detectors. Our key insight is that diffusion cross-attention enables semantic manipulation while preserving image structure. Based on this insight, we propose AMEND, a diffusion-based adversarial attack framework that integrates detector-derived conditioning, semantic attention dispersion, and structural consistency constraints. Experiments on seven object detectors show that AMEND achieves the lowest mAP on six of them, reducing relative mAP to as low as 6.73%, while consistently achieving superior visual naturalness across the evaluated settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial Attacks, Object Detection, Cross-Attention, Natural Adversarial Examples

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D2hHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article proposes a diffusion based attack image method, which has significantly improved performance compared to previous methods.

Critical strengths

The article differs from previous pixel-based attack methods by focusing on the cross-attention module in the diffusion model to weaken the detector's semantic relevance.

Critical weaknesses

Suggest further polishing or adjusting the writing style of the paper, focusing on the specific tasks of the article, without describing the background too much. It is recommended to start the article by stating whether the research focuses on attack methods or methods to solve attacks, in order to avoid ambiguity.

The author's insights are mentioned multiple times in the article, but the specific insights lack theoretical or experimental support, as seen in line 046 of the main introduction. It is recommended to use PDF format for the images in the article, which makes it easier to zoom in and view details, especially for the attack images studied in the article. There is too little related work content.

Questions for authors

Suggest adding visual content to the experimental results section to showcase the specific attack effects. It is difficult to demonstrate from the ablation results that the AMEND design achieves the optimal balance between attack performance and naturalness.

Can you evaluate the results of object detection methods specifically designed to handle images under attack? It is difficult to fully demonstrate the attack effect of AMEND by only providing the results of general detectors after being attacked by AMEND.

Reviewer_HB5xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMEND, a diffusion-based framework for category-specific adversarial attacks against object detectors. To balance attack effectiveness and visual naturalness, AMEND adopts a two-stage design that first performs DDIM inversion with unconditional-embedding optimization for faithful image reconstruction, and then optimizes perturbations in the latent space to generate adversarial images. Experiments on the INRIA Person dataset with seven object detectors show that AMEND achieves the lowest relative mAP on six of them and better visual naturalness in terms of LPIPS, SSIM, NIQE, and BRISQUE than the compared state-of-the-art methods.

Critical strengths
  1. The two-stage design separates faithful reconstruction from adversarial optimization. The first stage uses DDIM inversion and unconditional-embedding optimization to align the denoising trajectory with the inversion trajectory, improving reconstruction fidelity and providing a reliable latent for adversarial generation.

  2. The attack objectives (IoU-matched pre-NMS confidence suppression and cross-attention dispersion) effectively suppress target-category detections, while the fidelity components (self-attention consistency, MS-SSIM constraint, and attention-guided mask blending) limit structural distortion, achieving a good balance between attack effectiveness and image quality.

  3. AMEND is compared with several representative methods on seven detectors using the INRIA Person dataset, achieving the lowest relative mAP on six detectors with favorable LPIPS, SSIM, NIQE, and BRISQUE scores. Extensive ablations and comparisons at matched LPIPS levels further support its effectiveness.

Critical weaknesses
  1. Most components of AMEND build on existing techniques, and the paper does not clearly distinguish its own contributions from prior work.

  2. The baseline comparisons are insufficient, with limited coverage of recent adversarial attacks on object detectors, such as Diff-NAT [1].

  3. The evaluation is restricted to a single dataset (INRIA Person), without cross-dataset validation on the same target category, such as the person category in MS COCO.

  4. The choice of several key hyperparameters is neither sufficiently justified nor analyzed for sensitivity, including the timestep weights wtw_t in the self-attention consistency loss and the loss coefficients in the overall objective (Section 3.4), as well as the attention resolution and the number of unconditional-embedding optimization iterations NuN_u (Appendix A).

  5. The unconditional-embedding optimization is insufficiently described; in particular, its initialization, update strategy, and optimization settings are not clearly specified or justified.

  6. AMEND fails to achieve the lowest relative mAP on YOLOv5 and performs relatively poorly in YOLOv5-related transfer settings, but the paper does not sufficiently analyze or explain this result.

[1] Diff-NAT: Better Naturalistic and Aggressive Adversarial Attacks via Class-Optimized Diffusion for Object Detection. AAAI 2026.

Questions for authors
  1. Most components of AMEND, including DDIM inversion, unconditional-embedding optimization, and attention-based losses, build on prior work [2–4]. However, the paper does not sufficiently clarify how these techniques are integrated into a unified framework. What is the underlying rationale for this integration, and how does it differ from a straightforward combination of existing methods?

  2. The comparisons in Table 1 mainly involve earlier attack methods, while more recent diffusion-based attacks on object detectors, such as Diff-NAT [1], are not included. As a result, it remains unclear whether AMEND achieves state-of-the-art performance.

  3. The evaluation is restricted to the INRIA Person dataset, without cross-dataset validation on the same target category. Could the authors evaluate AMEND on the person category of other datasets, such as MS COCO, to demonstrate its generalizability across different scene complexities and image distributions?

  4. Figure 5 shows the trade-off between attack effectiveness and image quality under varying loss weights, and the matched-LPIPS comparison varies the number of attack iterations KK. Could the authors specify the exact loss-weight configurations used, and provide a sensitivity analysis of other key hyperparameters (e.g., wtw_t, the attention resolution, and NuN_u) to show how stable the reported performance is across different settings?

  5. The attention misalignment stage uses a 16×1616\times16 attention resolution, justified mainly by its ability to capture reliable object-level semantics. Could the authors explain why this resolution is preferred over others (e.g., 8×88\times8 or 32×3232\times32), and how this choice affects attack effectiveness and image quality?

  6. Several components of AMEND, including cross-attention variance minimization, self-attention consistency, and attention-guided mask blending, also appear in DiffAttack [2]. Could the authors clarify how AMEND differs methodologically from DiffAttack?

  7. Table 13 shows that AMEND outperforms the full-alignment variant in both attack effectiveness and image quality. Since the reconstruction stage builds on Null-text Inversion, could the authors explain why partial trajectory alignment is advantageous over full alignment, and how the choice of alignment steps affects the subsequent adversarial optimization?

  8. The reconstruction stage optimizes unconditional embeddings to align the denoising and inversion trajectories, but the optimization procedure is not described in sufficient detail. How are the unconditional embeddings initialized and updated, what optimization settings are used, and how are they applied across denoising timesteps?

  9. The self-attention consistency loss assigns a uniform weight (wt=1w_t = 1) to all denoising timesteps, although the discrepancy between clean and perturbed attention maps may vary across timesteps. What motivates this uniform weighting, and would a timestep-dependent weighting scheme yield better results?

  10. Tables 1 and 4 show relatively weaker attack performance in some settings where YOLOv5 is the source detector. What accounts for this gap compared with other source detectors, particularly in the white-box and cross-detector transfer settings?

[1] Diff-NAT: Better Naturalistic and Aggressive Adversarial Attacks via Class-Optimized Diffusion for Object Detection. AAAI 2026.

[2] Diffusion Models for Imperceptible and Transferable Adversarial Attack. TPAMI 2024.

[3] Denoising Diffusion Implicit Models. ICLR 2021.

[4] Null-text Inversion for Editing Real Images using Guided Diffusion Models. CVPR 2023.

Reviewer_3iKXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMEND, a diffusion-based adversarial attack framework that leverages attention mechanisms to balance attack effectiveness and visual naturalness for object detectors. Evaluated on seven detectors, AMEND achieves the lowest relative mAP on six of them. It also outperforms the compared methods in LPIPS and SSIM across all evaluated settings.

Critical strengths

Strong experimental results: AMEND achieves the strongest attack performance on six of seven detectors while improving visual naturalness. The improvements are particularly notable for YOLOv2, SSD, and Faster R-CNN.

Detector-specific design: AMEND combines an IoU-matched pre-NMS detection loss with a fixed attention-derived blending mask, providing an attack framework tailored to object detection.

Comprehensive experiments: The paper includes ablation studies, transferability evaluations, comparisons at matched LPIPS levels to examine the attack effectiveness–naturalness trade-off, and comparisons with DiffAttack-Det, a detector-adapted DiffAttack baseline.

Critical weaknesses

Limited novelty: The attention objectives adopt the same form as those in DiffAttack. The main contributions appear to be detector-specific adaptations, including detector-derived prompting, the pre-NMS detection loss, and the fixed blending mask. Furthermore, adding (L_{cross}} to (L_{det}) alone changes mAP by at most 1.86 percentage points, suggesting that the standalone contribution of attention misalignment is limited.

Limited evaluation: The experiments are restricted to 288 images from the INRIA Person dataset, a single target category, and relatively older detectors. Although the evaluation follows prior work by using clean-image detections as pseudo ground truth, performance against actual ground-truth annotations is not reported.

Fairness and efficiency: The attack settings differ between universal patch methods (NAP and AdvLogo) and per-image optimization methods (Diff-PGD and AMEND). Although matched-LPIPS comparisons are provided, comparisons under matched computational budgets are missing. Moreover, AMEND requires approximately 520 seconds per image, about 14 times longer than Diff-PGD.

Questions for authors

AMEND outperforms DiffAttack-Det in Table 13, but both methods use the same detection loss. The main differences appear to involve detector-derived prompting and the fixed attention-derived blending mask. What is fundamentally novel about attention misalignment itself compared with DiffAttack?

Could the authors provide additional evaluations using actual ground-truth annotations, including AP and recall?

The paper states that its detection loss can also be applied to DETR-style detectors. Does the full AMEND framework generalize to modern detectors such as DETR and multi-class datasets such as COCO?

Could the authors compare existing methods under matched computational budgets? Additionally, could they report the computational cost and attack effectiveness after implementing the efficiency improvements discussed in Appendix F, such as reducing unconditional-embedding optimization iterations or amortizing inversion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.