acceptodds
Withdrawn from ICLR 2027

Adversarial Examples, Inverted!

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Almost all adversarial attacks add an imperceptible perturbation to fool a model. We instead study the opposite: a large, clearly visible perturbation that causes the model to keep its original, correct prediction, even though a human would no longer recognize the image. We show such examples can be generated at scale, and then answer three questions this formulation leaves open: whether humans really perform worse than the model, whether standard out-of-distribution (OOD) detection and calibration tools catch it, and whether existing defenses mitigate it. We answer all three on MNIST, CIFAR-10, and ImageNet, and additionally verify that none of these answers depends on the specific attack variant or source architecture used to generate the stimuli.Throughout, we use NI-FGM (momentum-free, -normalized) as the primary attack and VGG-11 as the primary CIFAR-10 architecture, with a dense grid (12 points, up to 22 on CIFAR-10 and 13 on MNIST) and samples per track. Sec. sec:res-robustness reports a robustness check in which the entire pipeline is instead run with the momentum-based NMI-FGM attack and ResNet-18 as the source architecture, confirming every conclusion below is unchanged. (i) An independent recognizer proxy falls from near-ceiling to 20% on CIFAR-10 by the top of the widened range while the model stays at 100% — an 80-point gap, larger than under the original configuration, corroborated directly by a small human pilot () and not explained by signal loss (a matched-magnitude Gaussian control degrades recognizability faster); a CLIP zero-shot proxy confirms the gap at ImageNet scale too. (ii) Confidence- and energy-based OOD detectors and calibration are structurally blind (0% detection, ECE ), while a feature-space Mahalanobis detector flags 100% — but is evaded by an adaptive attacker at no cost to success. (iii) No classical defense, including adversarial training (45% robust accuracy), reduces attack success (correlation with large- resistance ). A mechanistic analysis further shows the attack destroys low-level texture far faster than edge/shape structure. Code, figures, and a ready-to-run human-study harness are included as supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Human-model perception gap, visible adversarial perturbations, model robustness, human recognition failure, out-of-distribution detection, confidence calibration, Mahalanobis detector, adaptive attacks, adversarial training, texture vs shape bias

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hpGrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate the opposite of adversarial attacks: a large, clearly visible disturbance that causes the model to retain its original, correct prediction, even though a human would no longer be able to recognize the image. They then answer three questions: whether humans actually perform worse than the model, whether common tools for detecting and calibrating out-of-distribution (OOD) data can detect this, and whether existing defense methods can mitigate this. The authors conduct various tests on three datasets.

Critical strengths
  1. The authors carry out many experiments. I also find the tests involving humans, as compared to models, interesting.
Critical weaknesses
  1. The paper is not well written or easy to understand. The abstract does not effectively introduce the topic and provides too many details right from the start (footnotes and references are also not useful there). The introduction really lacks an introduction to the topic as well as an explanation of the motivation behind the various steps of the method/experiments (right at the beginning, readers are presented with a formula without any explanation of the notation). Phrases such as “It does not, , and 4) Verify,” as well as the constant references to later sections, do not make for smooth reading. Section 3 is titled “Methods,” but seems to be more of an experimental setting. Various abbreviations, such as NKE, NI-FGM, etc., are used but not defined. The citations for datasets and networks are missing. Note, some tables extend beyond the edge.

  2. The novelty and timeliness of the approach seems unclear to me. The idea comes from the paper by Nguyen et al. (2015), what appears to be new is its application to ImageNet and a few other tests. The context, i.e., what the question is, what the task is, is not clear. Several aspects are being investigated, such as the distinguishability between humans and models, what appears appropriate, and shape/texture biases. On the other hand, tests are also being conducted using OOD methods, as well as adversarial-robust models and randomized smoothing, which are still unclear. Moreover, it feels like an outdated related work section (with the most recent source from 2019) from papers that has something to do with the topic being presented. For example, there are many newer OOD and randomized smoothing methods than those used in the paper.

Questions for authors
  • Are there any more recent works in this field or methods that have been applied? The most recent source is from 2022.
  • What is new in this study compared to the paper by Nguyen et al. (2015)?
  • I find the OOD definition vague, since it is still the same input image but with noise. So at what point should an OOD detection method classify the image as an OOD?
  • The idea behind adversarial training is that the models are more robust and do not change their predictions, but that is exactly what they are doing. Could you explain this point in more detail?
  • In Table 1, what are Models A and B?
Reviewer_Wp8uOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates a form of adversarial examples in which large, visible perturbations make images difficult for humans to recognize while preserving the model's original predictions. The authors propose a gradient-based method, NKE, to generate such examples and examine the resulting human–model recognition gap, cross-model transferability, and effectiveness of existing OOD detectors and adversarial defenses. Experiments on MNIST, CIFAR-10, and Imagenette further analyze the influence of attack variants, model architectures, and shape–texture characteristics.

Critical strengths
  1. The paper investigates the discrepancy between human perception and model predictions under large perturbations, and examines its implications for OOD detection, calibration, and adversarial robustness.

  2. The authors evaluate multiple datasets, model architectures, attack variants, OOD detectors, and adversarial defenses. The inclusion of a human pilot study, adaptive detection attacks, and shape–texture analysis provides additional perspectives on the investigated phenomenon.

Critical weaknesses
  1. The core phenomenon of human–model recognition discrepancies under large perturbations has been investigated in prior studies, including Excessive Invariance Causes Adversarial Vulnerability (ICLR 2019), Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations (ICML 2020), and Model metamers reveal divergent invariances between biological and artificial neural networks (Nature Neuroscience 2023). These studies have already examined model invariance, human perception, cross-model generalization, and adversarial robustness. The paper does not adequately discuss these works or establish sufficiently distinct scientific contributions.

  2. The evaluation retains images that preserve the source model's original prediction, raising concerns that the near-100% white-box success rate and ECE≈0 are partly consequences of sample selection rather than independent findings. Moreover, the human–model gap is primarily supported by other neural networks as recognizability proxies, while the actual human study includes only five participants. These limitations weaken the evidence supporting the paper's central claims about human perception, OOD detection, and adversarial robustness.

  3. Several statements lack clarity or consistency. For example, the term "asymmetry" is used to describe the white-box–black-box performance gap, whereas "limited cross-model transferability" would be more precise; directional transfer asymmetry should instead refer to differences between A→B and B→A transfer.

  4. Although the paper evaluates multiple aspects, including human–model recognition gaps, cross-model transferability, OOD detection, calibration, adversarial defenses, and shape–texture characteristics, the relative importance of these evaluation dimensions is not clearly established. As a result, it is difficult for readers to identify which results are central to the research question and which serve as supporting analyses. A clearer experimental hierarchy and stronger connections between the research objectives, evaluation metrics, and conclusions would improve the paper's coherence and readability.

Questions for authors
  1. How does the proposed NKE formulation fundamentally differ from prior studies on excessive model invariance and model metamers, particularly Excessive Invariance Causes Adversarial Vulnerability (ICLR 2019), Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations (ICML 2020), and Model metamers reveal divergent invariances between biological and artificial neural networks (Nature Neuroscience 2023)? What new scientific insights does this paper establish beyond these studies?

  2. Since the evaluation retains images that preserve the source model's original prediction, how do the authors ensure that the near-100% white-box success rate and ECE≈0 are not primarily consequences of sample selection? Can the authors report unconditional attack success rates on the original evaluation sets?

  3. Given that the main evidence relies on other neural networks as recognizability proxies and the human pilot includes only five participants, how can the authors establish that the observed gap genuinely reflects human perception rather than limited cross-model transferability?

  4. Since NKE explicitly minimizes classification loss and preserves the original prediction, why should ECE≈0 or the inability of traditional adversarial defenses to prevent label retention be considered evidence of their failure? Can the authors clarify the evaluation objectives and practical implications of these findings?

  5. The Introduction reports white-box success of 80–99% and black-box success below 5%, whereas Section 4.1 reports near-perfect white-box success and Table 1 shows considerably higher cross-model retention rates in several settings. Can the authors clarify the differences in experimental conditions and provide direct evidence supporting the claim that NKE transfers substantially worse than classical adversarial examples?

A clear and convincing response to these concerns could lead me to reconsider my final recommendation.

Reviewer_ZTnYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies large-perturbation adversarial examples that preserve a model's original prediction. It uses iterative loss minimization with an exterior L2 constraint and examines transferability, human recognizability, OOD detection, calibration, and conventional defenses. Experiments use MNIST, CIFAR-10, and ImageNet-pretrained models evaluated on Imagenette. The paper reports high source-model label retention, declining transfer across the evaluated settings, and different responses from confidence-based and feature-distance OOD detectors.

Critical strengths
  • The evaluation of the paper considers recognizability, transferability, OOD detection, and common adversarial defenses. This helps assess what the constructed examples reveal about model behavior.

  • The experiments include multiple attack variants, different source architectures, a matched Gaussian-noise control, and an adaptive attack against feature-distance detection. The paper also reports a small human pilot and acknowledges its limited size.

Critical weaknesses
  • Section 3.1 states that the evaluation retains images correctly classified both before and after perturbation. This raises a concern about the reported 100% source-model accuracy and near-zero ECE, since correctness is part of the selection rule and the attack minimizes true-label loss. It is also unclear whether attack-success and defense results use all attempted images or only retained examples. The paper should not treat low ECE on correctly classified, high-confidence images as independent evidence that calibration fails to identify perceptually unrecognizable inputs.

  • The novelty and human-recognition evidence are not sufficiently established. Prior work on excessive invariance and invariance-based adversarial examples, including Jacobsen et al. (ICLR 2019) and Tramer et al. (ICML 2020), is not discussed. The human pilot includes only five participants (260 judgments) and uses ResNet-18/NMI-FGM stimuli, whereas the primary experiments use VGG-11/NI-FGM. Most human-recognition conclusions therefore rely on other classifiers as proxies rather than direct human measurements.

Questions for authors
  1. How many initially correctly classified images were attacked, and how many were retained after the label-preservation condition? Please clarify the denominators for attack success, defense performance, and calibration, and report the relevant results before post-attack filtering.

  2. How is the proposed setting distinguished from the invariance-based adversarial examples in Jacobsen et al. (2019) and Tramer et al. (2020)? Which contributions are new relative to those studies?

  3. What evidence establishes that the large-L2 examples are out of distribution, rather than simply far from their particular source images? Large pixel-space distance alone does not establish that a sample lies outside the natural-image distribution.

  4. Can the human-recognition results be validated with more participants using the primary VGG-11/NI-FGM stimuli? For the matched Gaussian control, please also report the actual L2 distances after clipping, since matching before clipping does not guarantee equal final perturbation magnitudes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.