acceptodds
Under review as a conference paper at ICLR 2027

Which features remain vulnerable in robust models?

26 views

Abstract

Scaling adversarial training to larger models and hundreds of millions of synthetic images has improved robustness, yet these classifiers remain vulnerable to small input perturbations. What properties of their learned features make these attacks possible? We study 71 of the most robust CIFAR10 models publicly available, spanning a wide range of architectures and training conditions. Our analysis reveals that adversarial attacks manipulate the same features that are responsible for separating the classes on clean data. Moreover, this relationship is concentrated in a surprisingly small subset of features: for the most robust model, more than 70% of successful attacks can still succeed when restricted to just 10% of the features per class pair. Furthermore, we find that the samples that remain vulnerable can be identified as those which are poorly separated along these specific feature dimensions; and that successful attacks follow a consistent attack direction. In addition, at the model level, we demonstrate a gradient-free measure of margin spread that is strongly correlated with higher adversarial robustness. Together, these results provide new insight into the mechanisms that underlie the remaining vulnerability of highly robust classifiers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial examples, Adversarial robustness, Decision boundaries, Feature contributions, Representation geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mTceOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper provided theoretical analysis on what features are vulnerable against adversarial attacks on adversarially trained models. First, it proposes a gradient-free approach of using normalized margin separation for estimating the robustness of models. The paper then provides a feature analysis that shows adversarial displacements align with the features that separate class means on clean data. Lastly, the paper shows that this feature vulnerability is concentrated in a small subset of features.

Critical strengths
  1. The paper presents a theoretical analysis on the robustness of adversarially trained models, providing insight on why these models still break under adversarial attacks.
  2. The empirical scope of the paper is wide and thorough, spanning 71 models chosen from the RobustBench.
  3. The paper is well written.
Critical weaknesses
  1. Several works have been proposed to identify vulnerable features and to suppress or recalibrate them [1, 2, 3]. The paper needs discussion on how its notion of vulnerable features differs from those defined in these three methods. The paper would also benefit from extension on how their feature suppression or recalibration strategies can address the vulnerability identified in this paper.

[1] Bai et al., Improving Adversarial Robustness via Channel-wise Activation Suppressing, ICLR 2021

[2] Yan et al., CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection, ICML 2021

[3] Kim et al., Feature Separation and Recalibration for Adversarial Robustness, CVPR 2023

Questions for authors
  1. Do models trained with feature suppression or recalibration methods [1, 2, 3] exhibit the same concentration of vulnerability in class-separating features?
1 comment

We thank the reviewer for the positive feedback and for bringing relevant prior work to our attention.

Q1 / W1: Feature suppression and recalibration methods

These are good pointers, and we have added a discussion of CAS, CIFS and FSR in "Related work". Since models that use these methods are not currently on RobustBench they were not included in our analysis. That said, we expect that our observed concentration of vulnerability in class-separating features will occur for these methods, given the wide range of models in which we have seen such behavior. However, it is possible that their input-dependent channel gating may modify the class-separating features substantially, and we therefore propose the empirical investigation of this matter as future work.

Reviewer_j6R8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which features remain vulnerable even in highly robust classifiers. The authors analyze 71 CIFAR-10 models and show that the normalized margin separation correlates with model-level robust accuracy using only clean feature representations. They then study how individual penultimate-layer features contribute to classification margins and adversarial perturbations. Their results suggest that adversarial attacks tend to manipulate the same features that are important for separating classes, and that a relatively small subset of these features is attributed to attack success. Finally, they show that the most useful features retain much of the correlation between output margins and input-space margins, suggesting a connection between these features and the vulnerability of individual samples.

Critical strengths
  1. An interesting and relevant research question. I think it is worth investigating why even highly robust models remain vulnerable to adversarial attacks. Looking at the problem from a feature-level perspective is a reasonable approach, and I found the connection between class-separating features and adversarial vulnerability interesting.
  2. A fairly extensive empirical analysis. The authors examine 71 models with different architectures and robustness levels, providing the observations with a broader empirical basis. In particular, the finding that a small fraction of features can preserve much of the attack success is interesting and worth further investigation.
Critical weaknesses

While I find the empirical observations interesting, I am not fully convinced that they provide a substantially new explanation of adversarial vulnerability. The central finding that adversarial attacks exploit features that are also useful for classification seems closely related to the feature-based perspective of Ilyas et al. [1]. More recent work, such as Stevinson et al. [2], has also investigated adversarial vulnerability through feature geometry and interference.

I appreciate that this paper focuses specifically on highly robust models and provides a more detailed, quantitative analysis. However, it remains unclear how much of the observed behavior is specific to adversarial vulnerability, rather than a consequence of how the classifier uses a relatively small subset of feature dimensions to determine its output margin.

For example, the proposed normalized margin separation performs almost identically to the simpler normalized base margin (Table 3). Similarly, the strong results obtained using only the most useful features may partly reflect that these features already capture most of the original classification margin.

Overall, I think the experiments provide an interesting characterization of robust models, but stronger evidence is needed to establish what new mechanistic insight they offer beyond existing feature-based explanations and standard margin-based observations.

Questions for authors
  1. Novelty relative to existing explanations. I would appreciate some clarification on what the authors consider the main conceptual difference from Ilyas et al. [1]. Is the main contribution that the same feature-based explanation continues to hold for highly robust models, or is there a different mechanism underlying their remaining vulnerability? I am also curious how the findings relate to the representation-level explanation proposed by Stevinson et al. [2].

  2. Normalized margin separation (Section 3). In Table 3, the proposed metric achieves almost exactly the same correlation with robust accuracy as the simpler normalized base margin (Kendall's \tau of 0.5423 vs. 0.5382, with identical reported Pearson correlations). What additional information does the proposed metric capture? Given how similar the results are, I find it difficult to understand what is gained from the more elaborate formulation.

  3. Feature importance and vulnerability (Sections 4.2–4.3). Could the authors compare their feature ranking with simpler alternatives, such as rankings based on classifier weights or class-mean differences? It would also be useful to report how well the restricted output margin correlates with the full output margin. My concern is that the top features may simply preserve most of the original classification margin, which could already explain their strong correlation with the input-space margin.

  4. Dependence on the feature representation. The analysis treats individual penultimate-layer dimensions as distinct features. However, the importance and concentration of these dimensions may depend on the representation basis. For example, an orthogonal transformation of the features, together with a corresponding transformation of the classifier weights, would preserve the model's predictions but could change the feature rankings. Would the main findings remain similar under such transformations? An alternative feature decomposition, perhaps using sparse autoencoders, could also be informative here.

  5. Interpretation of feature interventions (Section 4.2). In the feature-fixing experiment, the resulting representations combine clean and adversarial feature values, but these representations may not correspond to any input within the allowed perturbation budget. I appreciate the additional gradient-masking and feature-flipping experiments in the appendix, but I am still unsure how far these interventions allow us to conclude about naturally occurring input-space attacks. Could the authors clarify this distinction or provide additional evidence connecting the results to realizable adversarial perturbations?

  6. Samples that do not follow the trend in Figure 3. I am not entirely convinced by the visual interpretation of Figure 3. While the overall trend is interesting, there seem to be quite a few exceptions. Some samples relatively far from their own class mean are not marked as vulnerable, while some vulnerable samples appear fairly close to their own class cluster. I understand that the figure only shows two feature dimensions, but I would be interested to know how common these exceptions are and whether they can be explained by the remaining features. More importantly, I think examining these cases could be particularly informative, as they may help clarify what aspects of adversarial vulnerability the proposed feature-based explanation can and cannot capture.

[1] Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019). Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32.

[2] Stevinson, E., Prieto, L., Barsbey, M., & Birdal, T. Adversarial Vulnerability from Interference Between Features in Superposition. Proceedings of the 43rd International Conference on Machine Learning

4 comments

We thank the reviewer for their suggestions, questions, and feedback. The additional ablations and analyses have improved the manuscript. We have reorganized the appendix to accommodate these additions, and all section references below follow the numbering in the revised manuscript. We welcome any further questions.

Q1 / W1: Novelty relative to Ilyas et al., standard versus robust models

We agree that the work of Ilyas et al. [1] is foundational, and consider ours to be an extension of theirs. Similar to Ilyas et al. [1], we find that it is the useful features that adversarial examples exploit, but we approach this from the very different perspective of output margin behavior, and focus on robust models, specifically. In this context, we make more specific claims regarding the behavior of adversarial examples, as outlined in our contributions. Furthermore, we compare with their measure of ‘feature usefulness’, i.e. correlation with the class label, in Appendices D.2 and D.3, and find that ours behaves differently: features can have substantial class label correlation (‘useful features’ as defined by Ilyas et al. [1]) while contributing little to the mean margin.

Our analysis shows clear differences between the robust models and the standard model we examine, and we believe the mechanisms to be different. We now show that the standard model falls outside the trend between output margin separation and robust accuracy (Appendix B.4), and that its output margins are much more sensitive to input perturbations (Figure 4, left, Appendix B.2). This is also supported by prior work such as Su et al. (2024), who find that targeted attacks can cause representations in standard models to ‘jump’ toward the target class cluster, while adversarially trained models retain much more of their clean class geometry. We have now reworked the related work section to make this clearer.

In terms of Stevinson et al. [2], we find less overlap. While they also relate adversarial perturbations to the geometry of the classification head, they focus on nonrobust models and ask how interference between class directions affects the logits of other classes during a targeted attack. This is a different focus from our work.

Q2 / W1: Normalized margin separation versus normalized base margin

The near-identical correlations are expected under the midpoint assumption. When mt=−mbm_t=-m_b, normalized margin separation mb−mt2sm,b\frac{m_b-m_t}{2s_{m,b}} reduces to the normalized base margin mbsm,b\frac{m_b}{s_{m,b}}. We now explicitly test this assumption in Appendix B.2 and indeed find that it holds, consistent with Table 3.

Although ‘Normalised margin separation’ offers little improvement in robustness prediction over ‘Normalised base margin’, we retain it because we decompose its numerator, mb−mtm_b-m_t, to determine how much each feature contributes to it in Eq. 15. This is our central measure of feature usefulness and connects the model-level analysis in Section 3 to the feature analysis in Section 4. In short: our theoretical analysis points to this quantity and it motivates how we determine which features adversarial examples are likely to exploit.

Q3-A: Comparison of feature ranking with simple baselines

This is an excellent suggestion, which we now explore in Appendix D.3. In short, we take many adversarial examples and calculate the adversarial contribution vectors for each sample (CadvC_{\mathrm{adv}}), which directly quantifies how much each feature contributed to the misclassification. We use this vector to obtain an oracle ranking and compare it with rankings based on our class mean contribution, absolute differences in classifier weights, and absolute differences between class means. Additionally, we also compare to the ranking given by the correlation with the label of each feature (from Appendix D.2). Our ranking has the highest mean agreement with the oracle across samples on each of the five models.

Q3-B / W1: Restricted versus full output margin

We now test this in Appendix D.4 and find that the restricted output margin strongly correlates with the full output margin. This is not surprising, since we rank features by how much they contribute to the difference in mean output margins between the two classes (Eqs. 13 and 15). This only reinforces our point that much of the separation between the two classes is captured by just a few features, especially since the feature orderings are calculated for a class-pair, and not per sample. Together with the feature fixing results from Section 4.2 we can conclude that separation along these few features already tells us which samples are more vulnerable, confirming our intuitions from Figure 3. We therefore do not believe this diminishes our claim that separation along these features explains which samples are vulnerable.

Q4: Dependence on the feature representation

This is an important insight: the basis dependence of our feature-based results stands in contrast to the rotational invariance of margin measures. We have therefore added a few sentences to the first paragraph of the section on "Feature Analysis", including a reference to some of the early work that motivated a preference for the particular basis found during training.

Q5: Interpretation of feature interventions

We agree that fixing features can produce representations that no input within the perturbation budget realizes. This is exactly why we consider the two additional masking and flipping experiments in the appendix. Of particular interest is the gradient masking one, as the resulting adversarial representations come from realizable input-space perturbations. In that case, we only limit the gradient information available to the attack. Furthermore, while each experiment has an artificial element, all three point to the same pattern and trends: adversarial misclassification depends strongly on a small subset of features, namely those that contribute most to class-pair separation. Also note that Table 1 provides further evidence for the latter using unmodified FAB attacks. We now make this more clear in the “Extensions” paragraph in Section 4.2.

Q6: Samples that do not follow the trend in Figure 3

Yes, we believe this is because only 2 out of 1024 features are shown, so samples that appear similarly placed in the figure can differ along the remaining features. This is best illustrated by Table 2, which directly quantifies how much additional features help rank samples by vulnerability. There is a clear jump in correlation with the input margin from the top-1% of features to the top-5%, indicating that features outside the top-2 can carry substantial information about vulnerability.

Complementary to this is Figure 2’s ‘useful first’ curve: while most attacks are unsuccessful after the top-10% most useful features are fixed, a smaller number (~18%) still remain vulnerable, indicating less reliance on these features. So there are certainly some samples that behave slightly differently than the majority, and would thus look like outliers when only the top-2 features are shown. We agree that examining the per-sample differences is a promising avenue for future work.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the effort the authors have put into addressing my questions. The new analyses have clarified most of my concerns and made the empirical findings more convincing. I am considering raising my score to 3.

My main remaining concern is about the scientific insight of the paper. While the experiments provide an interesting characterization of robust models, I am still not fully convinced how much new, vulnerability-specific insight they offer beyond existing feature-based and margin-based explanations. In particular, it remains unclear to me whether the observed concentration of adversarial vulnerability reflects something specific to adversarial robustness, or is largely a consequence of how ordinary classification-margin contributions are concentrated in a small subset of features.

I would appreciate further clarification on this point, as I think it is central to the paper's contribution.

Thank you for the further feedback and question. We hope the following addresses your concern.

We agree that the observation that a few features are mainly responsible for the clean output margin, by itself, would be a less interesting insight. However, our findings go further than this by showing that the features separating the classes are also those used by input-space attacks. This is perhaps best understood by considering Sections 4.1 and 4.2 together.

In Section 4.1, we show that the adversarial displacements align closely with the direction separating the two class means, when considering both feature and contribution space. This does not necessarily need to be the case: features that have a large contribution to the class separation could be insensitive to small input space perturbations, while features with little contribution could be highly sensitive to them. In addition, this class-mean direction is, by construction, shared between samples in a class-pair, as opposed to being sample-specific. The high alignment therefore shows that there is shared behavior in how adversarial examples exploit the various samples.

Section 4.2 then considers how many of the class-separating features are required. We find that successful attacks, per class-pair, depend on a small subset of them: most samples cannot be fooled when the top 10% of most useful features are fixed. The contributions to the mean output margins tell us which features contribute most to separating the classes on clean data, but not whether small input perturbations change those same features enough to cause misclassification. Our experiments establish this link.

Taken together, this result is interesting! The features which most contribute to separating the class-pair are exactly the ones exploited by the attack. Furthermore, as Figure 3 and Section 4.3 show: despite adversarial training on 300M training examples, there are still samples that are poorly separated in terms of the features that are most responsible for that separation. We believe this is a valuable insight for understanding why very robust models are still vulnerable, and how future defenses could intervene.

Compared to related work, Ilyas et al. also find that useful features are exploited in Standard models, but they do not establish that it is the most useful features that are preferred, nor do they establish the same class separation mechanisms and concentration we outline above.

Reviewer_AHkGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the features responsible for the remaining adversarial vulnerabilities of robust classifiers. The authors analyze 71 robust models from RobustBench on CIFAR-10 under ℓ∞\ell_\infty perturbations with ϵ=8/255\epsilon = 8/255. First, they propose a normalized margin separation score, motivated by a theoretical analysis, to estimate and rank model robustness without computing gradients. They then investigate how individual features contribute to adversarial misclassification. Their analysis shows that adversarial feature displacements are strongly aligned with the features responsible for separating classes on clean data. Furthermore, they find that a small subset of these features is sufficient for successful adversarial attacks, while restricting them significantly reduces attack success rates. Finally, they show that the same subset of features captures much of the relationship between output margins and sample-level adversarial vulnerability.

Critical strengths
  • The paper is very well written, easy to understand, straight to the point, and well organized.

  • The theoretical analysis of the proposed normalized margin is interesting and sound.

  • The feature contribution analysis provides interesting findings:

    • The relationship between the contributions of adversarial feature displacements that cause misclassification and the contributions of class averages.

    • Which features contribute to an attack and how many are needed.

  • Extensive evaluation on CIFAR-10.

Critical weaknesses
  • Novelty: The novelty of Contribution 1 is not well articulated. [1] and [2] have already proposed similar margin-based, gradient-free estimators. For example, your statement in L215 is not surprising, given that [1] established a relationship between worst-case robustness and the output margin: “We have shown that the robustness of the models we consider is closely tied to their output margin behaviour.” This relationship has already been established by [1] and [2].

  • Evaluation:

    • Lack of comparison with existing scores.

    • Lack of strong adversarial evaluation using AutoAttack. For example, the attack success rate claims in L366 should be verified using AutoAttack. Even if PGD is used for computational efficiency, its adaptive-step variant, APGD, is stronger in practice at an equivalent computational budget.

    • No evaluation on classification tasks with more classes and/or higher input dimension, e.g., CIFAR-100 and ImageNet-1k, despite abundant availability in RobustBench.

    • No validation with ℓ2\ell_2 norm

  • Individual experimental results are not provided. Table 1, Figure 2, and Figure 3 do not have extended results for other models in the appendix.

  • The theoretical analysis is mathematically sound, but the assumptions underlying Proposition 2 are insufficiently justified. In particular, the assumption that the decision boundary lies at the midpoint of the class means, as well as the approximate constancy of the normalized Lipschitz quantity across models and class pairs, are crucial for motivating the proposed score. However, their validity is not empirically established. Furthermore, although averaging the normalized margin separation across class pairs is reasonable, no theoretical result is provided to justify this aggregation for model-level robustness ranking.

References

[1] Ngnawé et al. (2024). Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers. NeurIPS 2024. https://arxiv.org/abs/2406.18451

[2] Li et al. (2024). GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models. NeurIPS 2024. https://arxiv.org/abs/2304.09875

Questions for authors

Q1: [1] demonstrates robust accuracy estimation using the logit margin (the difference between the top two logits), while Li et al. (2024) [2] use the GREAT score (a label-dependent version of the logit margin using a generative model) to rank models in RobustBench according to their robust accuracy. Can you compare your normalized margin score with these approaches for both robustness estimation using Algorithm 1 in [1] and model ranking, at least using the logit margin as a baseline? Note that both scores are also gradient-free.

Q2: Lines 208-209. You seem to imply that approximation quality depends on model robustness. Can you quantify this correlation? In [1], Ding2020MMA and Xu2023Exploring_WRN-28-10 are the models with the lowest margin consistency, yet they differ by more than 20 percentage points in robust accuracy. How does your score relate to margin consistency? Consider computing Kendall's tau correlation between the input margins and your normalized margins and comparing it with margin consistency.

Q3: Your analysis in Appendix A.2.1 relies on the PGD attack under the ℓ∞\ell_\infty norm, which tends to find perturbations at the maximum specified epsilon budget. Can you provide more details about this estimation? Why don't you use the FAB attack?

Q4: In the final row of Table 1, why do you consider only the non-robust samples and ignore those that are robust with respect to ε = 8/255?

Q5: To evaluate the number of features necessary for an adversarial attack, you apply the mask during the attack (Eq. 18). How does this compare to simply applying the mask to the adversarial features after a standard attack? Is this what you investigate in Section 4.3? If so, please clearly explain the similarities between the two analyses, as well as the differences in the conclusions they support.

Q6: Isn't the number of features necessary for a successful attack related to the number of features needed for a prediction, which is likely bounded by the rank of the linear classifier matrix, itself bounded by the number of classes? Evaluation on CIFAR-100 and ImageNet-1k will be particularly relevant here. A low-dimensional analysis of the features (e.g., using PCA) could also help provide more clarity.

Q7: In Line 377, you introduce the Jaccard index without explicitly defining it. While Appendix A.2.6 provides further analysis, the computation of the "top-20% feature Jaccard index" is not clearly explained. Could you provide a formal definition and clarify how to interpret the reported values? You may consider adding this clarification to the manuscript.

Q8: Can you empirically evaluate the validity of Assumption 2 across the 71 models? In particular, how close is (m_b + m_t) to zero, and how much does the normalized Lipschitz quantity (\gamma) vary across models and class pairs? Appendix A.2.1 measures feature sensitivity using successful PGD attacks, but this does not directly establish the constancy of the quantity required by Proposition 2.

Q9: In Line 303, you mention that "the train set samples are used to calculate the mean representations in all cases." Figure 3 also shows these mean representations. Could you additionally plot the corresponding means computed from the validation/test set? Please rule out potential distribution shifts between the training-set means, the full validation/test-set means, and the means of the validation/test subset used in the analysis.

References

[1] Ngnawé et al. (2024). Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers. NeurIPS 2024. https://arxiv.org/abs/2406.18451

[2] Li et al. (2024). GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models. NeurIPS 2024. https://arxiv.org/abs/2304.09875

7 comments

We thank the reviewer for the positive feedback, various suggestions, and questions. The additions have certainly improved the quality of the manuscript. To accommodate these, we have re-arranged the layout of the appendix, all section references are as currently numbered in the new version. We believe we have now addressed all major concerns, and welcome any further clarification.

Questions

Q1 / W1 / W2: Comparison with Algorithm 1, top-two logit margin, and GREAT score

TL;DR: We add a comparison with GREAT score; Algorithm 1 in [1] is not applicable.

While the scores are gradient free, Algorithm 1 in [1] is not a gradient-free evaluation approach, as it relies on a threshold found by computing the input margin on a small subset of samples. This evaluation method is not applicable for our normalized margin separation score, as ours is not a per-sample score, but rather per class pair, which makes applying Algorithm 1 difficult. This is also not the setting we consider, as the spread normalization and other design choices we make become unnecessary when a threshold calculation is allowed to calibrate the output margins.

We compare with the top 2 logit margin for model ranking, see the ‘Raw mean margin’ baseline in Appendix B.1 which is the mean of the top 2 logit margin. We have now also added a comparison with the GREAT score, which slightly underperforms ours (0.49 Kendall correlation versus our 0.54).

Q2-A: Approximation quality and model robustness

Yes, we have now added the following to the end of Section 3: “When removing the five least robust models and recalculating the Pearson correlation we find that it increases to 0.7967. Conversely, removing the five most robust models decreases it to 0.5830. This indicates that models which are not very robust behave differently. Our goal is to better understand highly robust models.”

Q2-B: Relation to margin consistency

We now investigate the link between our normalized margin separation and margin consistency in Appendix B.3. We find only a weak relationship (Pearson’s correlation of -0.337), which indicates that these two measurements capture different information. However, we also investigate the link between margin consistency and the sensitivity of the output margin to input perturbations, where we find a much stronger correlation of -0.667. This is a promising avenue for future work to further investigate when margin consistency holds.

Q3: PGD feature sensitivity estimate and FAB

We agree that PGD tends to find perturbations at the maximum epsilon budget, however the feature sensitivity formula divides by the size of the input perturbation ∥xadv−x∥∞\lVert x_{\mathrm{adv}} - x\rVert_\infty which accounts for this. We therefore believe PGD is an acceptable attack for this analysis. Furthermore, redoing this with FAB is incredibly expensive for the large number of models we consider. We reworked the appendix section (see response to Q8), and believe the updated analysis is more clear (now numbered Appendix B.2 and Equation 28).

Q4: Adversarially vulnerable and robust samples in Table 1

Table 1 now also includes a row for samples with margins larger than 8/255, and we have added to the text: “Furthermore, we compare samples which are adversarially vulnerable (mIS≤8/255m^{\mathrm{IS}} \leq 8/255) with those that are not (mIS>8/255m^{\mathrm{IS}} > 8/255) in the final two rows of Table 1. We find that these two groups show very similar alignment in both the ΔCMCb,t\Delta^{\mathrm{CM}}\mathcal{C}_{b,t} and Δb,tCM\Delta^{\mathrm{CM}}_{b,t} columns. This suggests that adversarial perturbations exploit similar class-separating directions for samples on either side of the epsilon bound.”

Q5-A: Applying the feature mask during or after an attack

Applying the mask at each iteration allows the attack to adapt to the fixed features. We now clarify in Section 4.2: “The mask is applied at each iteration so that the attack gradient is computed through the clamped representation, allowing the attack to adapt to the mask and search for a perturbation that succeeds with the selected features fixed. This formulation is therefore a more direct evaluation of whether the unfixed features suffice for a successful attack than applying the mask afterwards.” We consider applying the mask only after the attack in the ‘feature flipping’ experiment in Appendix C.1.

Q5-B: Difference between the analyses in Sections 4.2 and 4.3

Unlike Section 4.2, Section 4.3 does not consider an attack or adversarial representations. The method only considers the clean representations, masks some of the features, and then recalculates the output margin. This then allows us to determine how many useful features are needed such that the output margin correlates with the input margin - it is therefore a feature analysis of margin consistency, and allows us to determine which features mostly determine which samples are vulnerable. This is also now clarified in the text.

Q6 / W2: Feature count and number of classes

We now investigate this in Appendix C.2. We repeat the feature fixing experiment on four RobustBench models trained on CIFAR100 - which consists of 4,950 unordered class pairs, 110 times more than the 45 for CIFAR10. We find only minor differences to attack success rates, and the overall trends remain, indicating that the number of classes is not a significant factor.

Q7: Jaccard index and top-20% features

We now include the formal definition of the Jaccard index and the “top-20% features” in Appendix D.1, with an accompanying short clarification in the main text “...using the Jaccard index, which is the fraction of features in the union of the two sets that appear in both sets.”

Q8 / W4: Empirical validity of Assumption 2

We reworked the appendix (now numbered B.2) to directly test the assumptions using the quantities specified in Section 3: we estimate the mean γ\gamma and verify how constant it is across models, and similarly we investigate the midpoint assumption by measuring mb+mtm_b + m_t. We find that both assumptions hold to a large degree across the various models.

Q9: Train/test class means and distribution shift in Figure 3

Figure 3 now shows the test set means: we notice a slight drift from the train set means. During development we experimented with both the train set and test set means for calculating ΔCMCb,t\Delta^{\mathrm{CM}}\mathcal{C}_{b,t} and not found large differences in the various experiments. We therefore do not believe the slight shift is of concern to our results, and the train set is used throughout as it has the desirable property of not requiring a held out set to compute the various quantities.

Concerns not addressed in questions

W1: Novelty relative to margin-based estimators

We agree we have poorly articulated this. We now make the link to prior work more clearly: “We have shown that the robustness of the models we consider is closely tied to the separation of (normalized) mean output margins. This finding is consistent with prior work linking output margin behavior to robustness (Ngnawé et al., 2024; Li et al., 2024).” Our novelty lies in our specific formulation of the separation between normalized mean output margins and its correlation with model robustness. This also directly motivates Eq. 15, our central measure of feature usefulness, and therefore connects the model-level analysis in Section 3 to the feature analysis in Section 4.

W2: Stronger adversarial evaluation

We use PGD for computational efficiency, but have now performed an ablation using the stronger FAB for our fixed feature attack of Section 4.2, shown in Appendix C.3: we only find very small differences, where FAB has a slightly higher attack success rate for the ‘useful first’ curve, while the trends remain identical.

W3: Extended individual experimental results

Appendix E now shows the full results for Table 1 and 2 for all 71 models, as well as other selected models for Figure 2 and Figure 3.

W4: Justification of Proposition 2 assumptions

These are approximations of entities we are unable to measure (without utilizing gradients). The two underlying assumptions are (i) an approximately inverse relationship between function smoothness and margin spread, and (ii) a midpoint assumption: if the boundary is somewhere between two class means, we assume it’s in the middle, for simplicity. These are therefore not assumptions that we believe will also hold in general, but rather simplifications for the sake of analysis.

I thank the authors for their answers and for updating the manuscript.

- On Q1 / W1 / W2:

  • (1) I agree that Algorithm 1 in [1] uses gradients to determine a threshold on a small subset of samples. However, the goal of the comparison is not merely to find the winner but also to understand the scores. In this case, the question is whether the additional information they use yields better, similar or even lower approximations and rankings of the models' robustness.

  • (2) In the ranking comparisons of Appendix B.1, your defined raw margin mean does not use the logit margin, top 2 logits difference as defined by [1], i.e top class (correct or not) minus the largest competing one. As you mentioned, in your case, "Misclassified samples therefore contribute negative margins," which is not exactly the same thing. Could you rank the samples using the top 2 logits, regardless of the samples' correctness?

- ℓ2\ell_2 norm: Some validation for ℓ2\ell_2 norm (attack and robust models), even partial, would strengthen the contributions of this paper

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and discussion

We thank the reviewers for their careful reviews and a fruitful discussion period. The various suggestions have certainly improved the quality of the manuscript. We highlight the main additions below:

  • Extensions to the robustness ranking result. In Appendix B we now include additional baselines (top-2 logit margin and GREAT score), an extension to ℓ2\ell_2 models, empirical evaluation of our assumptions, and an analysis of a nonrobust model.
  • Extensions to the 'feature fixing attack' results, including results on CIFAR100 and an ablation using a stronger attack, both found in Appendix C.
  • A comparison of our method for feature ranking with simpler baselines in Appendix D.3.
  • An expanded related work section, including a discussion of methods that suppress or recalibrate vulnerable features.

Reviewer AHkG has confirmed that they have no other concerns, Reviewer j6R8 reported that these new analyses clarified most concerns and made the empirical findings more convincing, and Reviewer mTce reported no substantive concerns about soundness. We hope these additions and clarifications strengthen the case for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.