acceptodds
Withdrawn from ICLR 2027

DQ-PAD: Discriminative Quantized Prototype Alignment for Adversarial Patch Defense

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adversarial patches with diverse appearances challenge the generalization of defenses for object detection. Existing methods mainly localize adversarial patches by detecting statistical anomalies, learning discriminative patch features, or measuring reconstruction errors from generative models, but may struggle to distinguish diverse patches from benign regions. To address this issue, we propose Discriminative Quantized Prototype Alignment for Adversarial Patch Defense (DQ-PAD), which identifies adversarial patches by measuring how well image regions match benign visual patterns. DQ-PAD uses prototypes from a pretrained VQ-VAE2 codebook to represent benign visual patterns and computes an anomaly score based on the distance between each regional feature and its nearest prototype. However, since VQ-VAE2 is primarily trained for image reconstruction, some benign regions may match these prototypes poorly, while some patch regions may remain close to them. To address this limitation, we introduce patch-aware projection learning, which learns a projection matrix from labeled benign and patch regions while keeping the encoder and codebook frozen. The projection jointly maps features and prototypes into a low-dimensional space and selects more discriminative feature directions to reduce the matching distance of benign regions while enlarging the distance gap between patch and benign regions. In the pedestrian detection setting, we compare DQ-PAD with seven defense baselines using ten held-out adversarial patches and three object detectors, and demonstrate that DQ-PAD improves the average defense performance by up to 8.96% over existing SOTA methods. Physical-world experiments further demonstrate its applicability in real-world scenarios.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Patch Defense, Object Detection, Vector Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vpWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes DQ-PAD to address the limited generalization of defenses to adversarial patches with diverse appearances in pedestrian detection. DQ-PAD uses prototypes from a pretrained VQ-VAE2 codebook to represent benign visual patterns and measures regional anomalies through the distance between local features and their nearest prototypes. In addition, patch-aware projection learning maps local features and prototypes into a low-dimensional discriminative subspace. The evaluation includes INRIA, CCTVPerson, WiderPerson1000, and physical-world scenarios, with comparisons against seven defense methods using three object detectors and ten held-out adversarial patches.

Critical strengths
  1. Existing methods either rely on patch-specific training signals or introduce image distortions through generative reconstruction. This work detects anomalies through benign prototype matching, which may, in principle, facilitate generalization to unseen patches. The research problem has practical significance.
  2. By freezing the encoder and codebook, the method learns only a projection matrix. Its structure is simple, with low computational overhead.
  3. The work includes a range of evaluations and demonstrates competitive inference efficiency. The ablation study examines sensitivity to the hyperparameters.
  4. The manuscript is well structured and clearly organized.
Critical weaknesses
  1. In Section 5.4, the authors construct adversarial patches specifically targeting DQ-PAD to evaluate whether the system can still detect pedestrians when the attacker knows the defense. Figure 3 shows that these attacks have little effect on post-defense detection performance. However, the paper does not adequately specify the number of optimization iterations, whether different patch initializations were tested, or whether further optimization could strengthen the attacks. It is therefore difficult to determine whether the results reflect the effectiveness of the defense or insufficient optimization of the evaluated attacks. The authors should describe the patch generation and optimization procedures, test more initializations and larger optimization budgets, and examine whether the attacks can become more effective. They should then report detection performance under the strongest evaluated attack and provide examples of patches that are successfully defended against and those that cause missed detections, to clarify the conditions under which the method remains effective. In addition, the value reported in Table 3 is inconsistent with that in line 410 of the manuscript.
  2. What is “DQ-PAT”?
  3. The authors use the VQ-VAE2 codebook as a reference for benign visual patterns and learn a projection matrix to better distinguish benign regions from patch regions. However, the current experiments do not clearly establish the main source of the performance gains: the benign visual patterns represented by the codebook, dimensionality reduction, or a more discriminative representation learned using patch samples. The ablation results are insufficient to quantify the contribution of each design choice. Moreover, training involves three loss terms and hard-sample selection, but their individual contributions have not been evaluated.
  4. The paper uses one adversarial patch for training and ten held-out patches for testing to demonstrate generalization to unseen patches. However, generalization to an unseen patch pattern is not equivalent to generalization to a new attack method. For example, P1–P3 are generated by the same method, as are P4–P6; these patches may share similar features despite their different appearances. Furthermore, the paper does not specify which patch is used for training. It is also unclear whether comparable performance would be obtained with a different training patch. Thus, the current results demonstrate effectiveness against the evaluated patches, but do not yet provide sufficient evidence of consistent generalization to different types of unseen patches.
  5. The method identifies patch regions using a threshold: regions whose anomaly scores exceed the threshold are masked. The authors select a localization threshold of 0.1 and a projection dimension of 16, but do not clearly specify which data were used to select these hyperparameters.
  6. The authors observe a performance drop at higher projection dimensions and attribute it to retaining too much redundant information. However, changing the projection dimension may also change the scale of the anomaly scores. If the same localization threshold is used across dimensions, the performance drop may simply result from the threshold no longer being appropriate.
  7. The authors claim that the method localizes patches more accurately while reducing the removal of benign image content. However, this claim is primarily supported by pedestrian detection results and a small number of qualitative examples. Improved pedestrian detection performance does not directly establish more accurate patch localization. The paper also fails to adequately investigate whether normal content, including ordinary clothing patterns, complex backgrounds and non-adversarial stickers, may be misidentified as patches.
  8. The legends in Figure 6 are inconsistent with the plotted results.
Questions for authors
  1. For the adaptive attacks in Section 5.4, could the authors specify the number of optimization iterations, step size, and patch initialization procedure? Have they examined whether increasing the number of iterations or using different initial patch patterns can further strengthen the attacks? This information would help establish whether the results in Figure 3 are based on sufficiently optimized attacks. In addition, which value is correct, the one in Table 3 or the one in line 410?
  2. Which experiments establish the respective contributions of the VQ-VAE2 codebook, dimensionality reduction, and learning the projection matrix?
  3. Which specific patch is used for training, and which attack method generates it? Which test patches are generated by attack methods that are entirely excluded from training? Have the authors examined whether defense performance on other patches remains stable when a different training patch is used?
  4. Which data were used to select the localization threshold of 0.1 and the projection dimension of 16? In Figure 6, is the same threshold used for all projection dimensions? If so, what evidence rules out a mismatched threshold as a cause of the performance drop?
  5. Beyond pedestrian detection AP and qualitative results, are there direct measurements of patch coverage and the proportion of benign regions incorrectly masked to support the claim of more accurate localization and reduced removal of benign content? Have the authors observed substantial false positives on ordinary clothing patterns, complex backgrounds, or non-adversarial stickers?
  6. Do all occurrences of “DQ-PAT” refer to “DQ-PAD”? Please also clarify the correspondence between the legends and curves in Figure 6 so that the localization threshold and projection dimension results can be interpreted accurately.
Reviewer_6neBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DQ-PAD, a defense method against adversarial patches by identifying and removing the adversarial patches from the input images. It leverages a pretrained VQ-VAE2 codebook to present benign visual patterns and identifies adversarial regions by computing an anomaly score based on the distance between local image features and their nearest codebook prototypes. In addition, DQ-PAD introduces a learnable projection matrix that jointly maps image features and prototypes into a low-dimensional space to improve the discriminability of the anomaly score. The project matrix is trained with a hard-sample ranking loss, a benign feature compactness loss, and a feature preservation loss. During inference, regions with large anomaly score are identified and masked before object detection. DQ-PAD is evaluated against ten unseen adversarial patches on three object detectors. Experimental results demonstrate improvements over several existing defenses while maintaining inference efficiency.

Critical strengths
  1. The proposed method is simple and efficient. It leverages a frozen VQ-VAE2 codebook and only trains a low-dimensional projection matrix, which avoids the cost of training a large patch-localization network and achieves relatively efficient inference compared with several computationally expensive baselines.

  2. The proposed method demonstrates consistent performance gain compared to the baselines across Faster R-CNN, YOLOv3, RT-DETR models on the INRIA dataset.

  3. The evaluation covers multiple settings, including different object detectors, natural and non-natural adversarial patches, and physical-world scenarios.

Critical weaknesses
  1. The novelty of the proposed method is limited. The main technical contribution is a learned low-dimensional projection applied to codebook-matching residuals. While this is a simple and reasonable approach, the underlying components, including pretrained visual representations, nearest-prototype anomaly detection, metric learning, and hard-sample ranking, are well established.

  2. The experimental evaluation is mainly conducted on INRIA Person with additional results on CCTVPerson and WiderPerson1000. All three datasets focus on pedestrian detection, leaving the generalization of the proposed method to other object categories and more diverse detection scenarios unclear. Evaluations on broader benchmarks, such as COCO, would strengthen the paper.

  3. The proposed learning objectives in Sec. 4.3 includes three losses. There is no ablation study quantifying their individual contributions and validating the necessity of the proposed loss terms.

  4. The adaptive attack evaluation in Sec. 5.4 is ineffective. In particular, end-to-end adaptive attack substantially increases detection AP before defense compared with the original non-adaptive attack. This suggests the adaptive attacks are not sufficiently optimized and makes it difficult to judge the effectiveness of the defense.

  5. The physical-world evaluation does not show the object detection results before applying the defense, which makes it hard to judge the effectiveness of the proposed defense.

  6. The writing and formatting of the paper need to improve, including but not limited to inconsistent method naming (e.g., "DQ-PAT" instead of "DQ-PAD") and improper citation formatting.

Questions for authors
  1. In Figure 3, why do the end-to-end adaptive attacks become substantially less effective against the undefended detector compared to non-adaptive attacks?

  2. Can the authors provide a quantitative evaluation on physical-world attacks and additional object detection benchmarks beyond pedestrian detection?

  3. Can the authors provide an ablation study demonstrating the effectiveness of different components in Eq.(10)?

Reviewer_BKGdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DQ-PAD to localize and mask adversarial patches before object detection. It treats a frozen VQ-VAE2 codebook as prototypes of benign visual patterns and scores each local feature by its distance to the nearest prototype. Because the codebook is trained mainly for reconstruction, benign and patch distances can overlap, so the authors train only a projection that maps features and prototypes into a more discriminative low-dimensional space, pulling benign regions toward the prototypes and pushing patch regions away, then remove anomalous regions by a threshold.

Critical strengths
  1. The idea is clear. The method first represents benign visual patterns with a frozen codebook, then uses a lightweight projection to improve separation in a reconstruction space. Only the projection is trained. Localization depends on how well a region matches benign prototypes, rather than on memorizing one patch appearance.

  2. The main evaluation is broad. On INRIA pedestrian detection, ten patches held out from training and three detectors, Faster R-CNN, YOLOv3, and RT-DETR, are compared with seven defenses. The average AP exceeds the second-best method by 7.26, 8.96, and 5.91 points on the three detectors. The clean-image AP drop is within 1 point, and inference speed is reported.

Critical weaknesses
  1. The feature-layer study compares only the Top and Bottom levels of VQ-VAE2. Section 5.5 uses Figures 4 and 5 to argue that Bottom-layer features are more suitable for patch localization, but the only comparison is between these two levels. Intermediate encoder layers and other feature sources are not tested, so the Bottom level is only shown to be better than the Top level, not to be an appropriate localization feature in general;

  2. The three training losses are not ablated. The projection is trained with the ranking loss, the benign-region compactness loss, and the information-preservation loss, but no experiment removes any one of them. The paper also does not report AP for codebook matching without the projection;

  3. Section 5.4 repeatedly names the method DQ-PAT. DQ-PAT appears several times in the adaptive-attack section, including the white-box setting and the defense model in the end-to-end objective, while other parts of the same section use DQ-PAD. Please clarify whether this is a typo or a different method or variant.

Questions for authors

The same as the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.