acceptodds
Under review as a conference paper at ICLR 2027

ATLAS-AL: Adaptive Trust-Region for Latent Adversarial Searches via Active Learning

Abstract

Security evaluation of learning-based systems requires more than just testing the system against a fixed collection of attacks. It requires adaptive mechanisms that can efficiently discover sets of inputs that induce model failure. We introduce ATLAS (Adaptive Trust-Regions for Latent Adversarial Searches), which is a query-based framework that discovers adversarial input sets for black-box learning systems. ATLAS casts attack generation as an active learning level set estimation problem then combines calibrated approximations with a local-global sampling architecture to find regions of the input space that contain adversarial examples. Once discovered, ATLAS is designed to sample points within these adversarial regions to build adversarial sets that accurately represent the state of robustness of the target model. When applied on toy experiments, we find that ATLAS is able to recover more of the adversarial region under a limited query budget than does previous work. When applied to standard and adversarially trained MNIST, CIFAR, and ImageNet model targets, ATLAS produces better representative attacks than other query-based black-box attacks (NES, SignHunter, BayesOpt). ATLAS represents an automated red-teaming framework that can be used for both analyzing the robustness of learning-based systems under development and continuous auditing to see how the robustness of a system changes over time.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Machine learning, Bayesian Optimization, Active Learning, Level Set Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mbyTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ATLAS-AL addresses adversarial region estimation and sample collection under a limited query budget. It combines global and local Gaussian processes, residual correction, and complementary boundary and saturation sampling. Experiments on synthetic functions and image generators show promising results, particularly for synthetic region classification. However, the image evaluation leaves important questions about semantic validity, region estimation accuracy, and the interpretation of sample distribution metrics.

Critical strengths
  1. The joint objective is well motivated, and the ablations demonstrate a meaningful trade-off. Accurate region prediction and collecting many successful samples are distinct goals. In Table 2, saturation-only sampling on 100D Rastrigin achieves a 90.9% positive-query yield but an F1 of only 0.011. Mixed sampling improves F1 to 0.273, with a 50.0% yield. This provides empirical motivation for balancing boundary and interior sampling.
  2. The synthetic evaluation is systematic, with encouraging results under shared acquisition weights. The experiments cover three functions across three dimensions, using 30 seeds. In Table 3, ATLAS achieves the highest reported mean recall in 9/9 settings and F1 in 8/9 among the compared methods. With shared weights c=2,w=40c=2,w=40, Table 8 retains the highest mean recall in 8/9 settings and F1 in 7/9. Thus, these advantages do not depend entirely on adjusting these two weights separately for each benchmark.
Critical weaknesses
  1. The semantic validity of generated attacks is insufficiently established. Section 6 uses the generator’s source-class condition and the classifier’s target-class score to identify successful attacks, but conditioning does not guarantee that generated images retain the source label. For example, if a generator conditioned on “cat” produces an actual dog, classifying it as “dog” would not constitute misclassification. This is an illustrative possibility, not a claim about a specific reported image. Appendix E provides qualitative examples, but no independent annotation of source-label retention or semantically verified attack success is reported. Such validation is needed to interpret the reported ASR as evidence of classifier errors.

  2. The image experiments do not directly evaluate the learned region predictor. Unlike the synthetic experiments, Tables 4 and 9 report attack discovery and sample-set metrics without evaluating the final surrogate on an independent set of latent inputs. For example, a surrogate could correctly identify sampled attacks while incorrectly classifying an unqueried adversarial region. Statistics computed from discovered samples would not directly reveal this error. Held-out region classification and boundary-focused evaluation within the specified latent domain are needed to substantiate the region-estimation contribution in the image setting.

  3. The evidence for reduced clustering does not control for sample count. Equation (11) defines Spacing as mean nearest-neighbor distance divided by mean pairwise distance, which is sensitive to the number of points. For example, adding more equally spaced points to the same line segment reduces this ratio despite preserving uniform spacing. In Table 9, on standard CIFAR-10, ATLAS reports 71.8 samples with Spacing 0.63, whereas SignHunter reports 256.6 samples with Spacing 0.27. Consequently, the difference could partly reflect sample density rather than greater clustering. Repeated subsampling to equal set sizes within each source–target pair would help isolate distribution quality while retaining the original query-budget comparison.

  4. The proposed auditing interpretations lack controlled validation. Appendices E and F provide qualitative interpretations and discuss auditing applications, but do not independently test the proposed explanations. For example, Appendix E suggests that ocean backgrounds may explain why generated stingray images are classified as great white sharks. The examples alone do not establish whether the background, animal appearance, or other features drive these predictions. Controlled interventions, such as varying the background while preserving the subject, would help validate this hypothesis. Such evidence would strengthen the claimed diagnostic value beyond producing illustrative failure cases.

Minor Issue: Inconsistent calibration range (Table 6, p. 13). The caption identifies NC=50–100N_C=50–100 (8–15%) as a favorable range, but subsequently describes NC=112N_C=112 (17%) as “inside of this range.” Please correct this statement.

Figure and table placement. Table 4 and Figure 3 appear after the Conclusion, although they are discussed in Section 6. Placing them closer to the corresponding discussion would improve readability and the flow of the experimental presentation.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/2): semantic validity and region predictor

We thank the reviewer for an unusually precise review — each point identified something actionable, and all four led to changes in the paper. Our response spans two comments (1/2–2/2).

W1 — Semantic validity of generated attacks is insufficiently established.

"...conditioning does not guarantee that generated images retain the source label... no independent annotation of source-label retention or semantically verified attack success is reported."

We agree, and this was the most common concern across reviews. We added a section defining what counts as a successful generation and new analysis providing validation for the generated adversarial examples and also did analysis on ATLAS's ability to generate realistic images. For validation, we use held-out classification models and a realism metric to filter out examples with no valid semantic structure. With this filter applied, the reported ASR measures only successes on which the majority of held-out judges disagree with the target's misclassification and whose realism scores clear the threshold. For the additional analysis, we measure the distribution of realism scores in ATLAS's adversarial set compared to that of the images the generator is able to generate. Figure 7 shows us that the realism of ATLAS's images is capped by the quality generator it chooses to use. However, for both experiments we observe that ATLAS matches the cap. For instance, ATLAS matches the median realism for the BigGAN-256 generator.

[Revised paper] New Section 6.1 (Validating Generated Adversarial Images).

W2 — The image experiments do not directly evaluate the learned region predictor.

"...a surrogate could correctly identify sampled attacks while incorrectly classifying an unqueried adversarial region."

We agree this was missing. We moved the full tables to the appendix and added a result evaluating the surrogate in the image setting. Using ROC curves on held-out latents, we show the model learns to separate adversarial from non-adversarial regions for samples whose spread exceeds that of its originally collected set — i.e. the surrogate identifies adversarial examples far from its own adversarial set.

[Revised paper] New Figure 5 (region classification ROC and model gain) in Section 6.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/2): spacing metric, auditing, and minor points

W3 — The clustering evidence does not control for sample count.

"Equation (11) defines Spacing as mean nearest-neighbor distance divided by mean pairwise distance, which is sensitive to the number of points... Repeated subsampling to equal set sizes within each source-target pair would help isolate distribution quality."

This is correct, and we have adopted the reviewer's suggested fix. The metric (now called Unclustered) is computed by subsampling 20 times and averaging, using a sample size equal to the smallest adversarial set produced by any method on that source–target pair, capped at 400. This isolates distribution quality while retaining the original query-budget comparison.

[Revised paper] Eqs. (11)–(12) and Table 8 updated to the subsampled metric.

W4 — The auditing interpretations lack controlled validation.

We agree — the auditing interpretations are hypotheses, and a controlled study would be needed to validate them. We have softened the language in the auditing appendix to reflect this, and added edits clarifying that the audit observations are not the main takeaway. The main emphasis of the paper is the ability to generate synthetic adversarial datasets for target models, and how well the proposed setup produces such a set.

[Revised paper, Appendix E, "MEASURE"] "To validate such findings, we recommend conducting a follow up analysis, where controlled experiments (such as preserving scene context and cues while varying the subject or vice versa) validate whether or not the observations hold or if there is some point at which it breaks."

Minor 1 — Inconsistent calibration range (Table 6, p. 13).

Fixed in the revised PDF.

Minor 2 — Figure and table placement: Table 4 and Figure 3 appear after the Conclusion.

Fixed. We moved the table and the qualitative figures to the appendix and added representative plots next to the corresponding discussion in Section 6.

Thank you for the revisions. The equal-size subsampling addresses my concern about spacing, and the held-out evaluation is a useful addition. I also appreciate the more cautious wording of the auditing observations. I have two remaining questions: what fraction of the retained images are in the ambiguous group, and do the ASR and main comparisons remain similar when these images are excluded? Also, was the threshold in Figure 5(b) selected on data separate from those used to measure the gain?

Reviewer_cyuuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel method for discovering adversarial regions in the black-box score-based setting which is called ATLAS. They frame the problem of finding adversarial regions as level-set estimation problem. They demonstrate the method on toy examples and in the image classification problem comparing it to existing black-box classification score-based attacks on MNIST, CIFAR and ImageNet. The difference to existing methods is the focus on finding an adversarial set rather than a single adversarial example.

Critical strengths
  1. The paper proposes a novel approach with a specific target of finding adversarial sets for black-box models with classification score access which might be valuable for the adversarial robustness field since most methods focus on finding singular adversarial examples.

  2. Black-box attacks might be more practical than classical full access (white-box) attacks, therefore stronger methods for evaluating black-box model robustness are important for safety critical applications.

Critical weaknesses
  1. Evaluation quality. The main weakness that I see is the lack of comparison with Square Attack [1] which is the SOTA score-based black-box attack on image classifiers as far as I know. Without it the contribution of the paper to improving black-box robustness evaluation is not clear. The authors cite the Square Attack paper in Related Work, but the results are not included in Table 4 and I did not find any clarification why this comparison was omitted.

Multiple minor typos:

  • line 052: uses -> use;
  • line 130: solves -> solve;
  • line 280: measures -> measure.

[1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In ECCV 2020.

Questions for authors

The paper contains multiple notations for which I did not find a proper definition which negatively affects the clarity:

  • line 167: what is NcertN_{cert} ?
  • line 283: what is x∗x^* ? I did not find the definition in the main part and in the Appendix B where the details on toy functions are.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2

We thank the reviewer for the careful reading and for pointing us to the Square Attack comparison, which we have now added.

W1 — Missing comparison with Square Attack.

"The main weakness that I see is the lack of comparison with Square Attack [1], which is the SOTA score-based black-box attack on image classifiers..."

We initially left it out for consistency of representation with the other baselines. We have now added Square Attack to see how well its search mechanism curates adversarial sets relative to its peers and to ATLAS. We represent the latent sample as a single-channel 2D grid and restrict Square's search to the same restricted latent domain as the other baselines. Square finds successes the fastest, but its sets are less spread, more clustered, and its ASR is much lower on MNIST and CIFAR.

[Revised paper] New Section 6.1 (Validating Generated Adversarial Images); Square Attack results added to Table 8.

W2 — Multiple minor typos (L052 uses → use; L130 solves → solve; L280 measures → measure).

Fixed in the revised PDF.

Q1 — Undefined notation at L167 and L283.

Your markers rendered as question marks on our end, so we have inferred the intended symbols; please let us know if we guessed wrong.

  • L167 — we assume NcertN_\text{cert} was referenced. We now introduce it as a parameter input to the algorithm (L165).
  • L283 — we assume x∗x^{*} was referenced. We now describe it as the optimizer (L286).
Reviewer_F7EROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ATLAS, a query-based black-box framework for crafting sets of adversarial inputs that aims to discover the regions of the input space where adversarial examples lie and to sample new points there. To do so, an active learning level set estimation problem is solved using a surrogate model that learns which inputs are adversarial from the target model's outputs, which are then queried based on two different policies.

Critical strengths

None

Critical weaknesses
  • The paper's motivation and significance appear limited to me. ATLAS is framed as an adversarial attack, but practical evasion attacks start from given, existing, and valid input and aim to fool the model while preserving (leaving aside the amount of the applied perturbation) the initial point's content/meaning/validity/functionality (depending on the domain). In this case, synthetic data is generated with a GAN, without strictly applying and guaranteeing those constraints to the inputs. Indeed, there is no reliable check (such as human validation) to ensure that the generated image content is semantically valid, and Figs. 3-5 supports those concerns. The authors should clearly motivate their work, describing its practical use and the scenarios in which it applies. Moreover, the experimental evaluation also considers black-box evasion attacks as competing approaches, but it is not clearly described how they have been adapted to this setting. TRLSE and AP2L (the closest considered methods) are not evaluated on images, and TRLSE is not run with its optimal parameters, as shown in the original paper. Finally, from an adversarial machine learning perspective, several aspects of the proposed approach resemble query-based model extraction/stealing attacks (some of them even use active learning). The authors should position their work with respect to them and, if suitable, compare it with some of those approaches.

  • Assuming the goal of the proposed approach is model auditing only (which would anyway need a reframing), the provided insights do not seem novel and soundly validated. Several previous works have shown that deep neural networks rely on image background, context, textures, etc. Others, such as [a], proposed a method for discovering model failure regions. The authors should better frame their work with respect to the existing literature. Moreover, the provided insights are based on a small set of images and lack ablations or other sanity checks to validate the findings. Finally, the fact that adversarially trained models fail is not surprising, as they are expected to be robust to very small perturbations around in-distribution data.

[a] Eyuboglu, S., Varma, M., Saab, K.K., Delbrouck, J., Lee-Messer, C., Dunnmon, J.A., Zou, J.Y., & Ré, C. (2022). Domino: Discovering systematic errors with cross-modal embeddings. International Conference on Learning Representations. https://openreview.net/forum?id=FPCMqjI0jXN

Questions for authors
  • Could you please discuss the scalability to large datasets comprising several pairs of images belonging to several classes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/4): motivation and framing

We thank the reviewer for the detailed and constructive review. We respond to each point below, and indicate where the revised PDF has changed. Our response spans four comments (1/4–4/4).

W1 — Motivation and significance appear limited.

"The paper's motivation and significance appear limited to me."

We have added text to the introduction pinpointing the specific motivation driving ATLAS:

[Revised paper, L031–035] "As more learning-based systems get deployed in safety critical settings, generating large diverse adversarial data sets will be important for increasing robustness of system when real data collection and labelling becomes expensive. These sets can be used for adversarial training, augmentation, training an adversarial detector, and conducting deeper analysis about where and why these systems fail."

The motivation is centered on systems that extend beyond image classification models. Data collection and labelling is expensive for physical systems, and adversarial data — or any generic data that stress-tests the system — can be even more difficult to find. Any method able to generate synthetic adversarial data for such systems therefore has significant value. Generating these sets quickly is also meant to speed up the development of security defenses: adversarial datasets can be used for additional adversarial training, and also to train a detector for safety-critical systems. ImageNet-A [a], a popular adversarial dataset, is often used in exactly these ways.

[a] Hendrycks, D., et al. Natural adversarial examples. CVPR, 2021.

W2 — ATLAS is framed as an attack, but evasion attacks start from a valid existing input and preserve its meaning.

"...practical evasion attacks start from given, existing, and valid input and aim to fool the model while preserving ... the initial point's content/meaning/validity/functionality."

We believe there is a difference in premise here, and we have revised parts of the paper's framing to make it explicit. ATLAS is a framework that generates adversarial datasets, which is different from a framework that finds a single attack. The former has the broader aim of system robustification, requires a different problem formulation (which is what we argue for in the paper), and imposes additional requirements on what attacks are needed to construct a useful adversarial set — namely diversity and coverage, which we now highlight in the introduction.

[Revised paper, before L038] "Existing query-based black-box attacks are highly effective at finding an adversarial example per input with few queries. Therefore, a natural way to build adversarial sets is to rerun these attacks and pool their outputs. However, because each technique focuses on converging to a single point, repeated runs may rediscover similar failure modes."

[Revised paper, L038–043] "We argue that good adversarial sets should be diverse in its representation. It should either contain sets of input features that are diverse or inputs that achieve different levels of adversarial success. Therefore, we instead treat the adversarial region as the object of search, and ask: (i) whether a similar region-level search could recover larger and more diverse adversarial sets than pooled point attacks under an equal query budget..."

AuthorsOpenReview ↗

Response to Reviewer 1 (2/4): semantic validity and baselines

W3 — No reliable check that generated image content is semantically valid.

"...there is no reliable check (such as human validation) to ensure that the generated image content is semantically valid, and Figs. 3-5 supports those concerns."

This point was also raised by Reviewer 3, and we agree it is the most important issue to address. We have added semantic validation checks and analysis to filter out semantically invalid images. While a human validation study would be preferable, we implement an automated validation check. Specifically, we use held-out classification models (containing different architecture types) and a realism metric to filter out examples with no valid semantic structure. With this filter applied, the reported ASR measures only successes on which the majority of held-out judges disagree with the target's misclassification and whose realism scores clear the threshold. For the additional analysis, we measure the distribution of realism scores in ATLAS's adversarial set compared to that of the images the generator is able to generate. Figure 7 shows us that the realism of ATLAS's images is capped by the quality generator it chooses to use. However, for both experiments we observe that ATLAS matches the cap. For instance, ATLAS matches the median realism for the BigGAN-256 generator.

[Revised paper] New Section 6.1 (Validating Generated Adversarial Images); see also Figure 3 in Section 6 and Figure 8 in Appendix E.

W4 — Clearly motivate the work, its practical use, and applicable scenarios.

See our response to W1. We made additional edits in the introduction emphasizing the motivation to generate adversarial datasets that can be used to increase or assess the state of robustness of a system.

[Revised paper, L031–035] "As more learning-based systems get deployed in safety critical settings, generating large diverse adversarial data sets will be important for increasing robustness of system when real data collection and labelling becomes expensive. These sets can be used for adversarial training, augmentation, training an adversarial detector, and conducting deeper analysis about where and why these systems fail."

W5 — It is not clearly described how the black-box evasion baselines were adapted to this setting.

We have added this description to Appendix D:

[Revised paper, Appendix D] "We use the implementations of NES, SignHunter and Square Attack from BlackboxBench (Zheng et al., 2023) unmodified and use them to search over the restricted latent space. NES, SignHunter, and Square Attack are run in their L∞L_\infty form (where applicable), all starting around z=0z=0 and searching within a radius of ... Each iterate is projected back into this box after every step. Since Square perturbs image structures, each latent sample is transformed into a single-channel 2D grid; its search procedure is unchanged. Each step sets one window to ±ϵ\pm\epsilon and keeps the new proposed perturbation if the adversarial score improves. NES and SignHunter query probe points around their current samples to estimate their gradients/signs. These queries can grow outside of the restricted latent domain, so each query is clipped to stay within the allowed domain. ATLAS, BayesOpt, and uniform sampling query inside it by construction. NES, SignHunter and Square Attack run to the full budget, yielding their adversarial sets. BayesOpt is run in its native early-stopping configuration, since running it to the full budget is computationally infeasible given that its per-iteration cost grows with the number of observations (see Table 8). We therefore report ASR and queries-to-first-success for BayesOpt and omit its dispersion statistics, which require the run-to-budget adversarial set. Any duplicate latent samples selected are removed."

AuthorsOpenReview ↗

Response to Reviewer 1 (3/4): TRLSE/AP2L settings and model extraction

W6 — TRLSE and AP2L are not evaluated on images, and TRLSE is not run with its optimal parameters.

We did run TRLSE with its optimal parameters. From the original paper:

  • Appendix B.1: "VmaxV_\text{max} is often set to be from 1–10% of the search space's volume..."
  • Appendix B.1: "We keep β\beta fixed at 1.96 to demand higher confidence for all experiments."
  • Appendix D.1: "TRLSE generally works best with Vinit≈0.5dV_\text{init} \approx 0.5^d in practice, striking a local-global balance."
  • Algorithm 1: "if Vinit/2V_\text{init}/2, discard TiT_i."

All of these settings, and more, are implemented in our experiments (see Appendix A). Other settings were inherited from each method's original codebase for both TRLSE and AP2L, as we state in Appendix A: "We do not alter the baseline implementations, since doing so would require re-tuning them."

The only parameter that deviates from its original setting is the number of trust regions, which we fix to R=4R=4 to match ATLAS's configuration, so that the comparison stays equal and any difference reflects each method's search mechanism rather than its configuration. RR governs the number of simultaneously active regions in both methods, and reducing RR shrinks the overall candidate sampling pool for both methods without affecting their acquisition function, placement rule, or per-round sampling. The effect is therefore felt equally by both methods, and the matched setting does not favor ATLAS. We agree that a sweep over region counts for both methods would be informative, and we note it as a limitation and a direction for future work.

On evaluating TRLSE and AP2L on images: these two methods are the general-purpose search methods ATLAS builds on, and we included them to test one thing — whether ATLAS's search mechanism improves on theirs. The toy experiments are designed for that question, because they isolate search behavior from domain-specific factors. The image experiments answer a different question: how ATLAS performs in a realistic attack setting, for which the appropriate comparison is against other search methods designed to find such attacks, which we include. We also note that the dimensionality tested in the toy experiments (10–100D) spans that of our image experiments (32D), so the two should overlap in what they tell us. We agree that evaluating TRLSE and AP2L on the image tasks would be informative, and note it as future work.

W7 — Position the work with respect to query-based model extraction/stealing attacks.

Model extraction methods do from ATLAS substantially. Model extraction learns a surrogate that mimics the target's behavior, which requires different methodological framing from what ATLAS and similar query-based evasion attacks do. To make the distinction explicit, we add a sentence to the related work:

[Revised paper, Related Work] "Lastly, while some query-based model extraction methods... share our black-box query-limited threat model, their goal is to construct a surrogate that replicates the target model's behavior, while our work estimates an adversarial objective which is used to find adversarial regions of the input space."

AuthorsOpenReview ↗

Response to Reviewer 1 (4/4): auditing claims, related work, and questions

W8 — If the goal is model auditing, the insights do not seem novel or soundly validated.

We agree that the observation that models can condition on backgrounds is not novel in itself, and we do not claim these observations as the core contribution. The core contribution is an efficient search mechanism that generates adversarial sets for target systems, particularly for systems that need more adversarial data for training when data is scarce. The model auditing section is included only as an example of how such a set can be used. We have adjusted the language of the findings accordingly:

[Revised paper, Appendix E] "In this section, we give an example analysis of ATLAS' generated adversarial sets, which exemplifies one of many use cases for such a generated adversarial set (another which includes conducting adversarial training on the model). Note, this analysis only serves as an example of the type of findings one could make about their own target model through the generated adversarial sets."

W9 — Better frame the work with respect to existing failure-discovery literature.

The goal of methods like [a] is to learn a function that identifies a subset of existing data on which a target model fails. That approach is complementary to ours, but it is limited when existing datasets contain few inputs on which the target actually fails. For systems where data is already scarce, there is value in generating synthetic data that goes beyond what existing data provides. We add this distinction to related work:

[Revised paper, Related Work, first paragraph] "While these works are complementary to our methodology (we seek to generate new data), the limitation is that these approaches are ineffective when the existing dataset rarely contains inputs that make the model fail. In contrast, methods that generate valid synthetic data address this limitation, which is valuable, especially for systems in which data is already scarce."

W10 — The insights are based on a small set of images and lack ablations or sanity checks.

We agree the auditing interpretations are hypotheses that a controlled study would need to validate. As noted in W8, the audit in the appendix is not the core contribution — it is an example of the analysis one could do after generating such a set. We have softened the language and added an explicit call for controlled validation:

[Revised paper, Appendix E] "In this section, we give an example analysis of ATLAS' generated adversarial sets, which exemplifies one of many use cases for such a generated adversarial set... this analysis only serves as an example of the type of findings one could make about their own target model."

[Revised paper, Appendix E, "MEASURE"] "To validate such findings, we recommend conducting a follow up analysis, where controlled experiments (such as preserving scene context and cues while varying the subject or vice versa) validate whether or not the observations hold or if there is some point at which it breaks."

W11 — It is not surprising that adversarially trained models fail.

Please see our responses above, in particular W2 and W8.

Q1 — Scalability to large datasets comprising several pairs of images belonging to several classes.

The costs of generating large datasets scales linearly with the number of source-target pairs used in experiments. For ours, we use 20 source–target pairs per target model. Furthermore, because of the ambiguity that exists for images that could belong to multiple classes, we state a definition of a successful attack in Section 6.1:

[Revised paper, Section 6.1] "Since the main goal is targeted failures and since images can generally belong under multiple classes (ambiguity), we define a successful targeted failure as (a) an image generated under source class, (b) classified as the target class by the target model, (c) would not reasonably belong under the target class, and (d) that contains semantic features close to some real image."

Comments on the paper

1 comment
AuthorsOpenReview ↗

Feedback Addressed

We thank all reviewers for their detailed engagement. Two reviewers independently raised the issue concerning semantic validity of our generated images(R1, R3). We summarize the revision here and respond to each reviewer individually in their thread.

Semantic validity (main revision). We added Section 6.1, which validates every generated image with three held-out judges — chosen to differ in architecture from each other and from every target model (ResNet-18/ConvNeXt-Atto/ViT-Tiny for MNIST; VGG-19-BN/DenseNet-121/ Inception-v3 for CIFAR-10; ConvNeXt-S/Swin-B/ViT-B/16 for ImageNet) — together with a realism score computed from DINOv2-ViT-L/14 features (Kynkäänniemi et al., 2019). Every reported number now refers to the filtered set.

The resulting measurement is in Figure 7. On BigGAN-256, the median realism of ATLAS's adversarial set is 0.64, identical to the generator's own unconditioned samples (0.64) and to the images the target naturally misclassifies (0.64). Our images are therefore as realistic as that generator produces at all; the ceiling is the generator's, not the search method's. Table 8 reports realism per method, where ATLAS is comparable to every baseline under identical filtering.

Other changes.

  • New Figure 3 (p.9): qualitative ImageNet sets against standard and adversarially trained ResNet-50. These use BigGAN-deep, the setting where ATLAS's set falls furthest below its own generator's realism (median 0.67 vs the generator's 0.80; on BigGAN-256 the two coincide at 0.64, Figure 7).
  • New Figure 5 (p.10): ROC of the learned surrogate, evaluating the region predictor in the image setting (R3).
  • Square Attack added as a baseline throughout (Table 8) (R2).
  • The spacing metric now subsamples to equal set sizes, 20 draws averaged, removing its dependence on sample count (Eqs. 11-12) (R3).
  • Auditing language softened to present the interpretations as hypotheses requiring controlled validation (Appendix E) (R1, R3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.