acceptodds
Under review as a conference paper at ICLR 2027

Unveiling Adversarial Patches: Prototype-Guided Detection and Localization

Abstract

Adversarial patch attacks imperil modern vision systems because they are localized, physically realizable, and highly transferable. Existing defenses typically target either global-level detection (clean vs. patch) or patch localization, but most operate as black boxes and provide limited interpretability. We propose PatchTrace, a prototype-based adversarial patch defense that unifies global attack detection and patch localization within a single interpretable framework. PatchTrace learns to separate clean and anomaly prototype banks, detecting attacks by explicitly competing between normal and adversarial evidence. This dual-bank design enables highly accurate global-level detection of unseen patches and, via a novel Prototype Residual Attention Map (PRAM), it yields faithful, dense localization of adversarial regions. Unlike prior prototype-based defenses that remain restricted to global explanations, PatchTrace extends interpretability to the pixel level, supporting a coherent global-to-local defense pipeline. Extensive experiments demonstrate that PatchTrace achieves near-perfect detection on unseen patches while delivering stronger, interpretable localization than existing defenses, even in noisy settings. We assert that the proposed PatchTrace is the first prototype-driven, unified end-to-end framework that simultaneously achieves these four properties (detection, localization, interpretability, and robustness), and it can pave the way for a unified development of adversarial patch defense approaches in the future.

Primary area:
interpretability and explainable AI
Keywords:
Adversarial Patches, Prototype Learning, Adversarial Robustness, Interpretable AI, Patch Localization, Unseen Attack Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vVXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PatchTrace, a supervised framework for joint adversarial-patch detection and pixel-level localization. The method uses a ConvNeXt-Tiny backbone and two prototype banks representing clean and adversarial evidence. Global responses from the two banks are fused using Cross-Prototype Attention Fusion (CPAF) to determine whether an image contains an adversarial patch. For localization, the proposed Prototype Residual Attention Map (PRAM) compares anomaly and clean prototype responses spatially, and the resulting signal is decoded into a patch mask. The detector also gates localization so that a mask is produced only when an attack is detected.

The model is trained with image-level clean/attacked labels, ground-truth patch masks, prototype alignment and separation objectives, and dense localization losses. Evaluation includes cross-patch transfer from ImageNet-Patch to unseen patches on COCO, cross-dataset evaluation on SmuDPatch, GAP, noisy corruptions, and several physical-world datasets. PatchTrace reports strong detection and localization on ImageNet-Patch and SmuDPatch, although performance decreases substantially for broader attack distributions and some physical-world settings.

Critical strengths
  1. Unified and interpretable global-to-local formulation.

A major strength is that detection and localization are not implemented as independent modules with unrelated evidence. The same clean and anomaly prototype responses drive both the global attack decision and the PRAM localization signal. This makes the explanation more directly tied to the model decision than a purely post-hoc saliency mechanism and provides a coherent motivation for the architecture.

  1. Strong cross-patch localization and detection under the primary benchmarks.

The reported results on ImageNet-Patch and SmuDPatch are strong. PatchTrace reaches approximately 99.8% detection accuracy and 84.0% localization mIoU on the primary silent ImageNet-Patch evaluation, while also obtaining 99.1% detection accuracy and 70.6% mIoU on SmuDPatch without adaptation. The improvements over NAPGuard, DIFFender, and SAM-based baselines are substantial, particularly for localization.

  1. Good experimental analysis and computational efficiency.

The paper contains useful loss ablations, prototype analyses, noisy evaluations, OOD patch experiments, and computational comparisons. The ablations demonstrate that different parts of the objective affect localization and detection differently. The reported inference cost is also substantially lower than diffusion-based defenses such as DIFFender while maintaining much stronger localization performance in the reported benchmarks.

Critical weaknesses
  1. The robustness evaluation does not test an adaptive attacker or controlled cross-victim-model generalization.

The central claim is that the learned anomaly prototypes capture transferable adversarial-patch evidence rather than memorizing individual patch patterns. The paper convincingly evaluates transfer across patch patterns and, to some extent, attack-generation methods. However, I could not find an experiment where an attacker explicitly optimizes a patch to evade PatchTrace, for example by simultaneously reducing anomaly-prototype activation or increasing clean-prototype activation. I also could not find a controlled experiment where patches are generated against different victim architectures and then evaluated with the same PatchTrace model. Cross-patch transfer is not equivalent to cross-victim-model transfer. This is important because the learned prototype bank may partially capture artifacts induced by the attack generators or victim models represented in the benchmark rather than a model-independent notion of adversarial "patchness."

  1. Generalization deteriorates substantially in the broader and physical-world settings.

The near-perfect results on the primary benchmark do not carry over consistently to more diverse attacks. On GAP, which contains 25 patch types from eight attack methods, detection accuracy falls to 59.8%. On APRICOT, physical-world recall is only 23.7%. Furthermore, the physical-world subsets contain only attacked images, so these experiments cannot measure false positives on clean physical imagery. These results are especially important because the motivation of the paper emphasizes physical and safety-critical adversarial patches. The gap between the primary benchmark results and these broader settings weakens the claim of robust, general adversarial-patch detection.

  1. Localization relies heavily on explicit dense patch supervision, which weakens the interpretation that localization naturally emerges from prototype reasoning.

PatchTrace is fully supervised with ground-truth patch masks, and the ablation results show that removing the localization loss reduces mIoU from 90.4% to approximately 0.9%, while detection remains almost unaffected. Thus, the prototype-based detector alone does not appear to provide useful dense localization. The strong final localization performance is therefore heavily dependent on conventional pixel-level mask supervision and a learned decoder. This does not invalidate the method, but it makes the claim that prototype evidence itself provides faithful dense localization less convincing. It also raises practical questions about how the method would scale to attack families for which accurate patch masks are unavailable.

Questions for authors
  1. Clarification / material experiment: What victim model or models were used to optimize the adversarial patches in the ImageNet-Patch and SmuDPatch benchmarks? Are all patched samples guaranteed to successfully attack the corresponding victim model? Please clarify the attack-generation and filtering protocol, since this affects what the anomaly bank is actually learning.

  2. Material experiment: Can the authors evaluate cross-victim-model generalization? For example, train PatchTrace using patches optimized against one victim architecture and evaluate patches optimized against substantially different architectures such as CNN and ViT families. This would help determine whether the learned prototypes capture generic adversarial-patch properties or model-specific attack artifacts.

  3. Material experiment: How does PatchTrace perform against an adaptive attacker that knows the defense? In particular, an attacker could optimize its original attack objective jointly with a term that suppresses anomaly-prototype activation, increases clean-prototype activation, or directly minimizes the PatchTrace detection score. Since PatchTrace is proposed as a defense, this evaluation could materially change my assessment of its robustness.

  4. Clarification: The GAP detection accuracy is 59.8% and APRICOT recall is 23.7%, which are substantially below the main benchmark results. What characteristics of these attacks cause the failure? Is the degradation associated with naturalistic appearance, attack-generation method, victim architecture, patch scale, physical transformations, or another factor? A stratified analysis would help clarify the scope of the method's generalization.

10 comments
AuthorsOpenReview ↗

Responses and Thanks for the comments which further demonstrate the strength of the proposed work

1

Attack-generation protocol: ImageNet-Patch (Pattern Recognition'2023) uses adversarial patches generated against AlexNet, ResNet-18, and SqueezeNet, while SmuDPatch (ECCV'2026) uses ResNet-50, ViT, and MobileNetV2. Both are established, published benchmarks, and PatchTrace uses their existing adversarial patch samples without generating new attacks. Importantly, PatchTrace learns from the visual characteristics of adversarial patch regions rather than requiring every patch to fool its original victim model successfully.

2

Our evaluation already includes patches generated using diverse victim architectures: AlexNet, ResNet-18, and SqueezeNet (ImageNet-Patch), and ResNet-50, MobileNetV2, and ViT-B/16 (SmuDPatch). PatchTrace achieves strong cross-benchmark generalization without retraining, indicating that its learned representations transfer across different attack-generation settings. However, these benchmarks also differ in their patch-generation procedures, so this evaluation does not fully isolate the effect of victim architecture.

3. White-Box Adaptive Attack Evaluation

We thank the reviewer for this valuable suggestion. We conducted additional white-box adaptive attack experiments to evaluate PatchTrace against defense-aware adversaries.

Experimental setup: The attacker has full gradient access to PatchTrace and the victim ResNet-50. We optimize patch pixels using 200-step PGD, combining the targeted attack objective with three defense-aware objectives: (i) minimizing the detection score, (ii) suppressing anomaly and increasing clean prototype responses, and (iii) suppressing PRAM activation. We evaluate nine unseen COCO patches.

Table 3: White-box adaptive attack evaluation (%). Joint ASR represents attacks that both fool the victim and evade PatchTrace. The weight w controls the contribution of defense-evasion objectives.

Attack ObjectiveVictim ASRDetection RecallmIoUJoint ASR
Non-adaptive (reference)70.5100.090.40.0
Minimize detection score70.880.469.512.7
Suppress anomaly / boost clean prototypes70.575.264.816.3
Suppress PRAM activation70.795.286.14.3
Joint attack (w=0.1)70.689.279.77.7
Joint attack (w=1)70.367.559.021.7
Joint attack (w=10)66.044.734.036.2
Joint attack (w=1) + EOT32.683.373.72.9

PatchTrace maintains 95.2% detection recall against PRAM-targeted attacks and 89.2% detection recall when jointly optimized with w=0.1. Increasing the evasion weight to w=1 and w=10 produces joint attack success rates of 21.7% and 36.2%, respectively, illustrating the trade-off between attack strength and detection. Under EOT (noise and spatial shifts), PatchTrace retains 83.3% detection recall, with joint attack success decreasing to 2.9%. These results provide a more comprehensive assessment of PatchTrace under a strong white-box threat model.

4. Stratified Analysis

We conducted additional stratified analyses of APRICOT and GAP to identify the factors affecting generalization.

APRICOT: We analyzed all 873 physical adversarial images, examining patch contrast, scale, shape, victim architecture, target class, and imaging conditions.

Table 4: APRICOT detection recall across attack characteristics (%).

FactorDetection Recall
Patch contrast (lowest → highest quartile)6.4 → 37.0
Patch scale (smallest → largest quartile)10.1 → 32.0
Patch shape (square/circular)13.5 / 33.0
Victim architecture (3 models)19.4–32.2
Target class (10 classes)14.0–47.6
Perspective (flat/warped)22.9 / 26.3
Image sharpness (lowest → highest quartile)25.7 → 20.5

Patch contrast and scale show the strongest associations with detection performance (AUROC 0.69/0.63). Higher-contrast and larger patches are detected more reliably, while image sharpness and perspective warping have limited influence. Patch shape, victim architecture, and target class also contribute to performance variation

Controlled experiments further show that Gaussian blur (σ=4) reduces digital-patch detection from 100% to 0%, highlighting sensitivity to changes in patch appearance. On APRICOT, adjusting the operating threshold improves recall from 23.7% to 44.6% at 5% FPR, and to 59.5% with 2× multi-crop inferencewithout retraining

GAP: We additionally analyzed 1,942 annotated GAP images across patch appearance, scale, contrast, and sharpness. Unlike APRICOT, colour saturation is the strongest associated factor (AUROC 0.79): detection recall rises from 1.6% for low-saturation patches to 49.1% for high-saturation patches. Patch scale has a secondary, non-monotonic association, while sharpness shows limited influence. This suggests that muted, naturalistic patch appearances contribute to the generalization gap in the GAP

Overall, the analyses identify distinct challenges: physical patch contrast and scale on APRICOT, and low-saturation, naturalistic appearance on GAP.

Other/New Concerns and Discussion

We are actively working to resolve other concerns [if pending] and look forward to a positive assessment of our responses; we will be happy to address any new comments.

We have also strengthened our comparison by adaptively calibrating the attack detection thresholds of the existing approaches and demonstrated the\color{green}{\text{We have also strengthened our comparison by adaptively calibrating the attack detection thresholds of the existing approaches and demonstrated the}} $\color{green}{\text{computational efficiency of the proposed approach, as requested by R-jTVY.}} $

Thank you for your detailed rebuttal and the additional experiments. I sincerely appreciate the effort invested in addressing my concerns and providing further clarification.

The new adaptive attack experiments and stratified analyses are valuable additions that help clarify the strengths and limitations of the proposed approach. I will examine these results more carefully before reaching my final assessment.

Nevertheless, several important concerns remain:

  1. Novelty and methodological contribution: The approach primarily combines established components, including prototype-based representations, attention mechanisms, and supervised localization. I remain uncertain whether their integration constitutes a sufficiently substantial methodological advance over existing approaches in the literature.

  2. Generalization and robustness: The substantial performance degradation on GAP and APRICOT remains concerning, particularly given the paper's emphasis on transferable adversarial-patch detection. The additional analyses help explain these failures but do not fully address the underlying generalization limitations.

  3. Adaptive attacks: While I appreciate the newly introduced white-box experiments, the considerable reduction in detection recall under stronger adaptive attacks raises further questions about robustness against defense-aware adversaries.

  4. Dependence on dense supervision: The dramatic localization performance drop when removing the localization loss continues to raise questions about how much localization capability originates from prototype-based reasoning rather than conventional pixel-level supervision.

Overall, I appreciate the authors' responsiveness and the additional evidence. However, I believe these remaining issues are important for evaluating the paper's overall novelty, robustness, and significance relative to existing literature. I will carefully reconsider the rebuttal and the extent to which these concerns have been addressed before finalizing my assessment.

Reviewer_jTVYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PatchTrace, a defense against adversarial patches that unifies global image-level attack detection and local pixel-level localization of adversarial regions, not only for the sake of defense performance, but also for interpretability of the defense mechanism. PatchTrace is a learning-based approach that relies on banks of clean and adversarial image region prototypes. The proposed defense is evaluated in terms of image-level adversarial detection and patch-level localization across multiple benchmarks from the literature, including real-world data, and compared to existing baselines.

Critical strengths
  • The considered problem is relevant
  • The considered approach uses selective intervention (intervene only if attack is detected), which has been shown to perform well in recent works
  • Evaluation on real-world data is a merit
Critical weaknesses
  1. Evaluation: The main weakness of the paper is the inconsistency in the evaluations that are intended to support the main claims, especially in terms of the baseline defenses used for comparison. This critical weakness is broken down into specific points below.

1A) Fair comparison to baselines: the authors mention that methods that “require training” use the same training data as PatchTrace, however, it is not explicitly stated which baselines require training and which ones do not. Moreover, even for baselines that are not retrained, their detection thresholds could be calibrated using the same training data for a fair comparison. In fact, in Sec. D.5 in the Appendix, it is revealed that detection is performed with a threshold of 0.5, but it is not clear whether this is the default for all baselines in their original papers, and it is also not clear how this is would yield a more fair evaluation than allowing each detector to tune their threshold with the training data used by PatchTrace. On the same point, the collection of baselines is very inconsistent across evaluations, in particular NAPGuard is only used in Table 3, and despite being the strongest baseline in that table it is then omitted in Table 4. It also not clear why most baselines are omitted from the image-level detection results in Tables 1 and 2, and in particular given NAPGuard’s reported performance in their original paper, it raises the question of why it is omitted from the evaluation on the GAP dataset at the end of Section 4.2. The authors should provide a justification regarding the chosen baselines for each evaluation and more details on how they were trained or tuned to ensure a fair comparison. Moreover, in Sec. D.5 the authors mention they compute AUROC and average precision across attack probabilities (L771-772) but it is unclear if and how this relates to the main evaluations, as no further details or results are presented even in the Appendix. Finally, while the computational cost results in Section 4.5 and Appendix H are useful to understand PatchTrace’s gains in terms of computational overhead, once again the choice of baselines is not entirely justified, e.g., PAD is not even considered in any other evaluation, unlike SAM3 or AdvPatchXAI, whose computational cost is not mentioned.

1B) Explainability results: the authors state multiple times that interpretability is a key feature that distinguishes PatchTrace from existing defenses. While Appendix G adds numerical results that support this claim, it would be helpful to provide further insights into how PatchTrace compares to other detectors in terms of explainability. This would be especially relevant for defenses that already focus on explainability, such as AdvPatchXAI. It is unclear whether PatchTrace provides substantial improvements over existing approaches, either quantitatively or qualitatively.

1C) Adaptive attacks: it is well established in the literature that adversarial defenses can usually be evaded by white-box defense-aware attackers, and hence it is considered a key evaluation for non-certifiable adversarial defenses (SATED, [A], [B], [C], [D], [E]). However, there is no evaluation that considers this threat model, nor any discussion on its implications and how it may be a limitation of the current evaluation.

  1. PatchTrace description and distinction from earlier works: While Sections 2 and 3 aim to summarize how PatchTrace differs from existing defenses and its details, respectively, there are important details that should be addressed more carefully. See the specific points below for more details.

2A) Section 2 states that PatchTrace differs from earlier works by using an end-to-end prototype-based framework for global-level detection and pixel-level localization. However, to understand the relevance of these methodological/design choices, it is critical to understand how PatchTrace’s performance, interpretability, and selective intervention compare to existing work. Weakness 1 above already highlights some concerns with the first two of these points. Moreover, the importance of selective intervention has been addressed in the literature (SATED, [B]), but as no concrete victim models are considered in PatchTrace’s evaluation, it is unclear how its selective intervention compares to the corresponding existing work.

2B) Section 3 is focused on describing the main details of the proposed learning-based detector. While all relevant components and how they are incorporated in training are presented, inconsistent notation and sometimes incomplete information make it challenging to fully understand all the details of PatchTrace (e.g., F’fF’_f, F’cF’_c, PkbP^b_k, and SbkS^k_b are all introduced in L185-192 without clear descriptions or details on their dimensionality, in (11) the numerator takes a summation over r≠sr\neq s without introducing ss nor rr, and it is likely that another outer summation over ss is missing). This concern is aggravated by the fact that as stated later in the reproducibility statement, the paper’s descriptions are intended as the main source for future implementations and reproductions of PatchTrace.

References: [A] Croce, F. et al. “Evaluating the Adversarial Robustness of Adaptive Test-time Defenses”. In Proc. of ICML, 2022. [B] Byrd Victorica, M. et al. “Saliuitl: Ensemble salience guided recovery of adversarial patches against CNNs”. In Proc. of CVPR, 2025. [C] Lin, Z. et al. “I Don’t Know You, But I Can Catch You: Real-time Defense Against Diverse Adversarial Patches for Object Detectors”. In Proc. of ACM CCS, 2024. [D] Tarchoun, B. et al. “Jedi: Entropy Based Localization and Removal of Adversarial Patches”. In Proc. of CVPR, 2023. [E] Rossolini, G. et al. “Defending from Physically-realizable Adversarial Attacks through Internal Over-activation Analysis. In: Proc. of the AAAI, 2023.

Questions for authors

Please address the points raised in Weakness 1. For Weakness 2A, how does PatchTrace differ from existing works, in particular, what are its key advantages in terms of selective intervention and explainability? How does PatchTrace perform against an adaptive adversary?

2 comments
AuthorsOpenReview ↗

Responses and Thanks for the comments which further demonstrate the strength of the proposed work

Baseline Fairness and Calibration

Training protocols: AdvPatchXAI, NAPGuard, and Kumar & Agarwal (2026) are trained using the same data as PatchTrace. DIFFender and SAM 3 are evaluated as pretrained, training-free baselines without fine-tuning.

Calibration protocol: We calibrated DIFFender (63 configurations) and SAM 3 (54 configurations) using only the Patch-1 ImageNet training split. We optimized prompts, inference thresholds, and mask selection rules for localization. Since neither method has a dedicated attack-detection head, we also calibrated image-level detection thresholds using their output scores. All configurations were evaluated on nine unseen COCO patches, using identical images and noise realizations. Computational costs were measured on an NVIDIA H200 GPU.

We want to mention why the calibration was not performed earlier:\color{green}{\text{We want to mention why the calibration was not performed earlier:}} it is to note that these algorithms are claimed to be generalized and handle unseen\color{red}{\text{it is to note that these algorithms are claimed to be generalized and handle unseen}} patch attacks; therefore, to check their true potential, the calibration was not performed. As the uncalibrated model shows these existing approaches are\color{red}{\text{patch attacks; therefore, to check their true potential, the calibration was not performed. As the uncalibrated model shows these existing approaches are}} generalized and robust, the success of the proposed algorithm becomes prominent as it does not require any calibration for unseen patches, noise,\color{red}{\text{generalized and robust, the success of the proposed algorithm becomes prominent as it does not require any calibration for unseen patches, noise,}} and datasets. However, based on the suggestion of the reviewer, we have now performed additional experiments by carefully calibrating the threshold\color{blue}{\text{and datasets. However, based on the suggestion of the reviewer, we have now performed additional experiments by carefully calibrating the threshold}} of the existing approaches.\color{blue}{\text{of the existing approaches.}}

Table 1: Calibrated baseline comparison (%, except parameters and latency).

MethodSilent mIoUNoisy mIoUSilent F1Noisy F1ParamsLatency
DIFFender (default)16.113.666.7*66.7*1066M78.8 ms
DIFFender (calibrated)26.222.866.264.81066M78.8 ms
SAM 3 (default)37.815.353.127.6841M56.6 ms
SAM 3 (calibrated)89.166.788.975.2841M56.6 ms
PatchTrace90.380.4100.099.333M5.4 ms

*DIFFender's default rule predicts every image as attacked, yielding the trivial all-positive F1 of 66.7%.

Calibration substantially improves SAM 3 localization (37.8% → 89.1% Silent mIoU), bringing it close to PatchTrace under Silent conditions. Nevertheless, PatchTrace retains four advantages:

  1. Noise robustness: PatchTrace achieves 80.4% Noisy mIoU, exceeding calibrated SAM 3 by 13.7 points, with a substantially smaller degradation under corruption.
  2. Detection: PatchTrace achieves 100.0%/99.3% F1 (Silent/Noisy), compared with 88.9%/75.2% for calibrated SAM 3.
  3. No threat-specific prompting: SAM 3 requires a text prompt and top-1 instance selection, whereas PatchTrace operates without these assumptions.
  4. Efficiency: PatchTrace uses 25–32× fewer parameters and is approximately 10–15× faster than these baselines.

Baseline Selection and Computational Efficiency

We have expanded the computational comparison to include AdvPatchXAI and SAM 3, alongside DIFFender and PatchTrace, and clarified the inclusion of PAD. PAD evaluation: We already report PAD's overall average localization mIoU in the main paper. To further address the reviewer's concern, we additionally provide its patch-wise mIoU on the COCO patched dataset under the Silent setting. PAD achieves an average mIoU of 45.5%, consistently below PatchTrace across all evaluated patches.

Table 2: PAD localization performance (Silent mIoU, %).

PatchP1P2P3P4P5P6P7P8P9P10Avg.
PAD45.345.845.645.546.444.844.645.346.045.245.5

Updated computational comparison:

MethodParameters (M)Latency (ms)Relative Latency
DIFFender1066.078.814.6×
SAM 3840.556.610.5×
AdvPatchXAI23.517.5 ± 3.43.2×
PatchTrace (ours)33.45.41.0×

Latency is measured on an NVIDIA H200 NVL at 224×224 resolution, batch size 1, using 50 timed runs after warm-up.

PatchTrace is 3.2× faster than AdvPatchXAI, despite having more parameters (33.4M vs. 23.5M). AdvPatchXAI employs 2,048 prototypes with spatial prototype matching and clamping, whereas PatchTrace uses 384 prototypes with cross-bank attention fusion. PatchTrace also achieves 10.5× and 14.6× lower latency than SAM 3 and DIFFender, respectively.

These additional results provide a more consistent evaluation of computational efficiency across representative prototype-based, foundation-model-based, and diffusion-based defenses. We will update Section 4.5 and Appendix H accordingly.

AuthorsOpenReview ↗

[Resiliency, Success, and Deployability] Adaptive Attacks and Experimental Consistency

Adaptive Attack

We conducted additional white-box adaptive attack experiments to evaluate PatchTrace against defense-aware adversaries.

Experimental setup: The attacker has full gradient access to PatchTrace and the victim ResNet-50. We optimize patch pixels using 200-step PGD, combining the targeted attack objective with three defense-aware objectives: (i) minimizing the detection score, (ii) suppressing anomaly and increasing clean prototype responses, and (iii) suppressing PRAM activation. We evaluate nine unseen COCO patches (200 images each).

Table 3: White-box adaptive attack evaluation (%). Joint ASR represents attacks that both fool the victim and evade PatchTrace. The weight w controls the contribution of defense-evasion objectives.

Attack ObjectiveVictim ASRDetection RecallmIoUJoint ASR
Non-adaptive (reference)70.5100.090.40.0
Minimize detection score70.880.469.512.7
Suppress anomaly / boost clean prototypes70.575.264.816.3
Suppress PRAM activation70.795.286.14.3
Joint attack (w=0.1)70.689.279.77.7
Joint attack (w=1)70.367.559.021.7
Joint attack (w=10)66.044.734.036.2
Joint attack (w=1) + EOT32.683.373.72.9

Key findings: PatchTrace maintains 95.2% detection recall against PRAM-targeted attacks and 89.2% under joint optimization with w=0.1. Increasing the evasion weight to w=1 and w=10 produces joint attack success rates of 21.7% and 36.2%, respectively, illustrating the trade-off between attack strength and detection. Under EOT (noise and spatial shifts), PatchTrace retains 83.3% detection recall, with joint attack success decreasing to 2.9%. These results provide a more comprehensive assessment of PatchTrace under a strong white-box threat model.

Notation and Reproducibility

We have revised Section 3 to provide explicit definitions, dimensions, and consistent indexing throughout. Feature and prototype dimensions: For a 224×224 input, the projected fine and coarse features have dimensions 768×14×14 and 768×7×7, respectively. The prototype-space features have 256 channels, with 256 clean and 128 anomaly prototypes. We now explicitly define the spatial similarity maps, pooled prototype responses, and bank representations at their first occurrence. Equation (11): We clarify the ground-truth mask index set, explicitly define the spatial summation and normalization, and include the omitted outer summation over batch samples. Notation consistency: We standardize the use of spatial indices, prototype indices, bank identifiers, and normalization symbols throughout Section 3. We also include a notation table in Appendix A that specifies the meaning and dimensionality of each symbol. These revisions resolve the identified ambiguities and make the mathematical formulation more precise and reproducible.

Baseline Consistency

We thank the reviewer for highlighting this concern. We clarify that our original evaluation selects the strongest relevant baselines from the initial comparison for further evaluation on SmuDPatch. To address the concern regarding NAPGuard's omission, we conducted additional evaluations on SmuDPatch under both Silent and Noisy settings.

NAPGuard on SmuDPatch: Table 4: NAPGuard cross-patch localization on SmuDPatch (mIoU, %).

SettingP0P1P2P3P4P5P6P7P8P9Mean
Silent36.830.549.239.948.447.932.917.142.628.737.4
Noisy16.813.627.423.623.318.65.57.423.716.217.6

Results are averaged across all 10 training checkpoints for each SmuDPatch evaluation split.

NAPGuard achieves 37.4% Silent mIoU, which decreases to 17.6% in the presence of noise, indicating limited generalization in cross-patch localization. Its corresponding image-level detection F1 scores are 65.6% (Silent) and 35.0% (Noisy). These results further support PatchTrace's stronger generalization to unseen adversarial patches.

Baseline selection and GAP evaluation: We distinguish between image-level detection and pixel-level localization when selecting baselines. We adapted NAPGuard for pixel-level patch localization, whereas GAP does not provide the precise pixel-level masks required for an equivalent evaluation of localization. This explains its omission from GAP localization comparisons.

Other/New Concerns and Discussion

We are actively working to resolve other concerns [if pending] and look forward to a positive assessment of our responses; we will be happy to address any new comments.

$\color{green}{\text{We have also strengthened our approach by adding experiments requested by other reviewers, such as stratified analysis and physical-world success.}} $

Reviewer_W85uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PatchTrace, a framework for detecting and localizing adversarial image patches. It learns separate clean and anomaly prototype banks, combines their responses through Cross-Prototype Attention Fusion (CPAF), and uses a Prototype Residual Attention Map (PRAM) to guide a segmentation decoder. The detection decision gates the final patch mask. The evaluation trains on ImageNet images containing one source patch per run and tests on unseen patches in COCO images, SmuDPatch, GAP, and physical-world datasets. It also examines additional image corruption, component losses, prototype activations, and inference cost.

Critical strengths
  • The paper connects image-level patch detection with pixel-level localization through shared clean and anomaly prototype evidence. The prototype visualizations and inference-time anomaly-bank ablation provide some evidence that these responses influence the detector, rather than serving only as post hoc explanations.

  • The evaluation covers held-out patch patterns, a change of image dataset, different attack-generation methods, and noisy inputs. On the ImageNet-Patch cross-patch protocol, evaluated using COCO test images, the method reports 99.8% detection accuracy and 84.0% localization mIoU in the silent setting. The reported 8.95 ms inference time is also considerably lower than DIFFender's 120.68 ms, although these methods are measured at their native input resolutions.

Critical weaknesses
  • The contribution of the proposed architectural components is not sufficiently isolated. Table 5 ablates training losses, and Appendix F removes anomaly prototypes at inference, but neither directly tests CPAF, PRAM, or the dual-bank design against simpler alternatives. Removing localization supervision reduces mIoU from 90.4% to 0.9%. Since the segmentation decoder also receives backbone features, the reviewer fails to understand how much improvement comes specifically from PRAM rather than conventional supervised segmentation.

  • Generalization is weaker on the more diverse evaluations. Detection accuracy falls to 59.8% on the balanced GAP dataset, and recall is 23.7% on APRICOT. Localization mIoU on noisy SmuDPatch is 34.8%, compared with 70.6% without added corruption. The physical-world datasets contain only patched images, so they do not establish false-positive behavior. The experiments also do not test benign image overlays or attacks adapted to PatchTrace. These gaps limit the broad robustness claims.

Questions for authors
  1. How does PatchTrace compare with a supervised segmentation decoder using the same backbone features but no PRAM? Can the paper separately ablates CPAF and the dual prototype banks? These comparisons would clarify which parts of the architecture account for the reported gains.

  2. What explains the drop on GAP and APRICOT? Can the detector distinguish adversarial patches from visually similar but benign overlays, and how does it perform against patches optimized with knowledge of PatchTrace? These are important tests of the claimed generalization.

  3. Section 3.3 states that the detection gate is applied after decoding, while Appendix E.1 says the localization branch is not triggered for clean inputs. Does the segmentation decoder run on every input? Please clarify the actual inference path and how it is reflected in the reported runtime.

4 comments
AuthorsOpenReview ↗

Responses and Thanks for the comments which further demonstrate the strength of the proposed work

1. Component-Wise Ablation Analysis

We thank the reviewer for this suggestion. To isolate the contributions of PRAM, CPAF, and the dual-bank design, we conducted additional component ablations (Table 1). Each variant replaces one component while keeping the backbone, training protocol, and losses unchanged unless specified. All models are trained on Patch 1 and evaluated on nine unseen patches under Silent/Noisy settings.

Table 1: Component ablation (F1 and mIoU in %).

VariantSilent F1Silent mIoUNoisy F1Noisy mIoU
Full PatchTrace100.090.299.480.2
PRAM ablation
Backbone features only (no PRAM)100.088.998.376.0
PRAM only (no backbone features)100.089.699.079.3
Thresholded PRAM (no decoder/Lloc)100.033.399.431.6
Thresholded feature-norm saliency100.07.899.410.0
CPAF ablation
Equal-width MLP100.088.897.470.4
Self-attention (no cross-bank attention)22.254.599.376.8
Dual-bank ablation
Single unassigned bank99.282.266.545.4
Anomaly bank only99.987.897.267.2
Single bank + alignment/tanh66.784.166.774.3

PRAM vs. supervised segmentation: We agree that dense supervision substantially contributes to localization. However, removing PRAM reduces mIoU by 1.3/4.2 points (Silent/Noisy), while PRAM alone retains 89.6/79.3% mIoU. Without the decoder or Lloc, thresholded PRAM achieves 33.3/31.6%, compared with 7.8/10.0% for feature-norm saliency. Thus, PRAM provides meaningful spatial evidence beyond conventional supervised segmentation, particularly in the presence of noise. The 0.9% mIoU without Lloc in Table 5 reflects the lack of decoder supervision, not the absence of prototype evidence.

CPAF: Replacing CPAF with an equal-width MLP reduces Noisy F1 by 2.0 points and mIoU by 9.8 points. Self-attention without cross-bank interactions severely degrades Silent F1 (100.0% → 22.2%), supporting the importance of structured cross-bank fusion.

Dual-bank design: With 384 total prototypes across variants, a single unassigned bank reduces Noisy F1 by 32.9 points and mIoU by 34.8 points. An anomaly-only bank loses 13.0 mIoU points under noise. Clean prototypes provide reference evidence for distinguishing anomalies from benign content. Applying the original alignment/tanh losses to a single bank further degrades detection.

Overall, these ablations confirm that PRAM improves spatial localization and robustness to noise, that CPAF enables effective prototype fusion, and that dual-bank modelling is critical for robust generalization.

2. Physical Generalization and Benign Overlays

We conducted additional experiments to investigate real-world generalization, and benign overlays.

1. APRICOT generalization: We evaluated 873 physical adversarial images and 873 clean images. Patch size, image resolution, and physical appearance contribute to the performance drop. Recall increases with patch area, but APRICOT's average patch size (3.0%) lies within the training range (1.2–5.0%), indicating that size alone does not explain the gap.

Table 2: APRICOT detection analysis (%).

MetricSingle-view2× Multi-crop
Recall (threshold = 0.5)23.731.2
Recall at 5% FPR44.659.5
AUROC0.780.93

The 5% FPR operating point and multi-crop inference improve recall to 59.5%, demonstrating the importance of threshold selection and spatial resolution. These results also highlight sensitivity to physical image transformations.

2. Benign overlays: We evaluated 1,800 COCO images with overlays matched to adversarial patches in size, shape, and location.

Overlay TypeDetection Rate (%)
Real adversarial patch100.0
Shuffled adversarial pixels98.9
Random noise64.9
Blurred adversarial patch (σ=4)0.0
Natural-image overlay0.0
Solid-color overlay0.0
Clean image0.0

PatchTrace distinguishes adversarial patches from benign natural-image overlays, rather than simply detecting occluded regions. The results suggest sensitivity to high-frequency patch statistics, helping to explain part of the difficulty with physical transformations. Random-noise responses also identify a boundary of this discrimination.

3. Inference Procedure and Runtime

Inference path: PatchTrace runs the detector and segmentation decoder on each input, then suppresses the mask to obtain clean predictions. All reported experiments follow this procedure.

Runtime: On an NVIDIA H200, detection takes 6.06 ms, decoding 0.84 ms, and complete inference 7.00 ms. The decoder contributes only 12% of total latency. Skipping decoding for clean inputs can further reduce computation without changing predictions.

We will correct Appendix E.1 and clarify the inference procedure and runtime in the revised manuscript.

Other/New Concerns and Discussion

We are actively working to resolve other concerns [if pending] and look forward to a positive assessment of our responses; we will be happy to address any new comments.

We have also strengthened our comparison by adaptively calibrating the attack detection thresholds of the existing approaches and demonstrated the\color{green}{\text{We have also strengthened our comparison by adaptively calibrating the attack detection thresholds of the existing approaches and demonstrated the}} $\color{green}{\text{computational efficiency of the proposed approach, as requested by R-jTVY.}} $

$\color{blue}{\text{We have also strengthened our approach by adding experiments requested by other reviewers, such as stratified analysis, physical-world success, and adaptive attack.}} $

Thanks for the rebuttal response. My concerns are largely addressed. I am happy to push the score by 1 level.

We are happy to know that our responses addresses the concern raised and appreciate the increase in rating to the acceptance level.

Thanks

Comments on the paper

6 comments
AuthorsOpenReview ↗

Discussion and Acknowledgement of our Responses

We sincerely thank the reviewers for providing constructive comments even with these hectic deadline schedules.

But as we are aware that we’re about halfway through this critical discussion period, it would be great if you can revisit our response and provide us positive acknowledgement or provide us the timely opportunity to resolve any new/remaining comments.

3. PAD Localization Under Noise

We thank the reviewer for this suggestion. We have now included PAD's complete patch-wise results under both Silent and Noisy settings and provide a detailed quantitative and qualitative analysis in Appendix I (Section: Why PAD Improves Under Noise?).

Table 5: Patch-wise localization comparison (mIoU, %).

MethodSettingP1P2P3P4P5P6P7P8P9P10Mean
PADSilent45.345.845.645.546.444.844.645.346.045.245.5
PADNoisy60.164.462.961.165.261.659.163.766.364.762.9
PatchTraceSilent88.090.286.175.890.184.387.881.581.474.884.0
PatchTraceNoisy72.880.269.139.680.865.873.457.457.947.464.4

Quantitative findings: PatchTrace substantially outperforms PAD under Silent conditions (84.0% vs. 45.5% mean mIoU) and retains a higher mean mIoU under noise (64.4% vs. 62.9%). We acknowledge that PAD achieves higher Noisy mIoU on four patches (P4, P8, P9, and P10).

Understanding PAD's unusual Noisy performance: Interestingly, PAD's mean mIoU increases from 45.5% to 62.9% under noise. We investigate this behavior in detail in Appendix \ref{sec:PAD_diag}, including qualitative visualizations. Gaussian noise changes PAD's fused heatmap responses and SAM mask-selection behavior, reducing clean-setting over-segmentation and producing masks that overlap more closely with the ground-truth patches. Representative examples show IoU increases from 0.420 to 0.920, 0.580 to 0.846, 0.289 to 0.514, and 0.444 to 0.632. These observations suggest that PAD's improvement under noise arises partly from the interaction between heatmap thresholding and SAM overlap matching, rather than establishing uniformly improved robustness. The additional analysis provides a transparent comparison, including cases where PAD outperforms PatchTrace. While PAD is competitive under noise, PatchTrace maintains higher average localization accuracy in both settings, with substantially stronger Silent performance and lower computational overhead.

4. Adaptive Attack Trade-off and EOT Analysis

We agree that our initial results (w10) do not establish a clear trade-off, as increasing the evasion weight consistently improves Joint ASR with only a modest reduction in victim ASR. To investigate this further, we extended the optimization to w=30,100,300,1000 and evasion-only (w=) and conducted additional EOT experiments.

Table 6: Extended white-box adaptive attack evaluation (%).

Defense Weight (w)Victim ASRDetection RecallmIoUJoint ASR
0 (Non-adaptive)70.5100.090.40.0
0.170.689.279.77.7
170.367.358.921.8
1066.245.034.036.1
3060.838.926.640.1
10052.036.123.539.8
30044.835.122.539.6
100035.134.622.034.1
(Evasion only)0.134.121.90.0

Attack–detection trade-off: Joint ASR increases with the evasion weight and peaks at 40.1% (w=30). Beyond this point, stronger evasion optimization reduces victim ASR without improving Joint ASR. Thus, the trade-off emerges at higher weights than initially evaluated. Even under evasion-only optimization, PatchTrace retains 34.1% detection recall, indicating residual detection resistance under the evaluated 200-step PGD budget.

EOT investigation: We further examined the unexpectedly low performance of the initial EOT attack. Applying the same EOT transformations without defense evasion also reduces victim ASR to 33.0%, indicating insufficient attack optimization rather than stronger defense robustness. Increasing the optimization budget to 600 PGD steps with four EOT samples per step raises victim ASR to 78.3%, reduces detection recall to 34.2%, and achieves 50.0% Joint ASR. The resulting patches also retain 81.7% victim ASR under test-time noise and spatial shifts, compared with 1.4% for non-EOT adaptive patches under identical transformations.

These experiments clarify two important findings: (i) stronger defense-aware optimization eventually encounters a trade-off between victim misclassification and detection evasion, and (ii) sufficiently optimized EOT attacks pose a more substantial threat than our initial experiments suggested. We will replace the earlier EOT results and explicitly report 50.0% Joint ASR as the strongest observed adaptive attack, while clarifying the remaining robustness limitations of PatchTrace.

5. NAPGuard Global Detection Evaluation

We thank the reviewer for emphasizing the importance of consistent baseline evaluation. Although NAPGuard does not have a dedicated image-level clean-versus-patched classification head, we additionally evaluate its global detection performance on GAP and APRICOT using the same clean and patched images as PatchTrace.

Evaluation protocol: NAPGuard produces patch mask with associated confidence scores. We derive image-level detection from these outputs: an image is classified as patched if at least one predicted patch exceeds the detection confidence threshold; otherwise, it is classified as clean. We then compute recall on patched images, FPR on clean images, and balanced accuracy. Both methods use their Patch-4-trained checkpoints, without retraining on GAP or APRICOT.

Table 7: Global detection comparison (%).

DatasetMethodRecallClean FPRBalanced Accuracy
GAPPatchTrace20.50.959.8
GAPNAPGuard3.20.651.3
APRICOTPatchTrace23.70.861.5
APRICOTNAPGuard45.00.972.1

PatchTrace achieves higher balanced accuracy on GAP (59.8% vs. 51.3%), while NAPGuard performs better on APRICOT (72.1% vs. 61.5%). Both maintain low false-positive rates. These results provide a direct comparison of their image-level detection capabilities and highlight their complementary generalization strengths.

While NAPGuard showcase better performance (only) on APRICOT, it is to note that there are several factors including explainability, robustness, and joint defense, which makes the proposed algorithm highly suitable for real-world deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.