acceptodds
Under review as a conference paper at ICLR 2027

Interpretable Adversarial Defense via Class-Aware Neuron Sensitivity Attribution and Recalibration

Abstract

Research on analyzing and enhancing model robustness at the neuron level has attracted significant attention due to its interpretability. However, existing methods are either based on neuron sensitivity evaluated by changes in activation values, or based on neuron importance measured by contributions to clean samples. Neither approach considers the intrinsic connection between the changes in neuron activations and the neural network's outputs. To bridge this gap, we propose a novel Neuron Sensitivity Attribution (NSA) method. By jointly calculating the dual contributions of intermediate neurons in suppressing the true label and promoting the adversarial label, this method provides a fine-grained, category-specific evaluation of neuron sensitivity under adversarial perturbations. Based on this metric, we propose a Sensitivity-Aware Neuron Recalibration mechanism to reform adversarial training strategies. Specifically, during adversarial training, stronger penalties are assigned to highly sensitive neurons, thereby enabling the model to rely more heavily on robust neurons. Extensive experiments across various models and adversarial training frameworks demonstrate that the proposed method can serve as an effective plug-and-play module to further enhance the robustness of deep learning models.

Primary area:
interpretability and explainable AI
Keywords:
Model interpretation, adversarial examples, neuron sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8PMROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates adversarial robustness at the intermediate-neuron level. The authors propose Neuron Sensitivity Attribution (NSA), which applies DeepLIFT to paired clean and adversarial inputs and scores each neuron by subtracting its attribution to the adversarial-class logit from its attribution to the ground-truth-class logit. These class-wise scores are then used in Sensitivity-Aware Neuron Recalibration (SANR), a weighted feature-alignment regularizer incorporated into adversarial training. On CIFAR-10, neuron-masking and Grad-CAM analyses are conducted with ResNet-18 and VGG-16. Integrating SANR into AT, TRADES, and DKL generally produces modest improvements in adversarial robustness.

Critical strengths
  1. The paper targets a relevant gap in activation-change-based neuron sensitivity analysis. Measuring how changes in intermediate neurons relate to the true-versus-adversarial class decision offers a useful class-conditional perspective on adversarial vulnerability, beyond activation magnitude alone [2].
  2. SANR is a straightforward regularization component compatible with multiple adversarial training objectives. Table 1 shows improvements on both tested architectures when it is added to AT, TRADES, and DKL, including under PGD-100 and AutoAttack rather than only a single attack.
  3. The authors examine a meaningful limitation of hard masking. Section 4 and the class-wise visualizations show that removing neurons identified as sensitive can help certain attacked inputs but also substantially reduce clean accuracy, which motivates a softer training-time intervention.
Critical weaknesses
  1. The novelty of NSA relative to existing attribution methods is insufficiently established. Section 3.1 and Equation (4) compute the difference between two DeepLIFT attributions of the same clean/adversarial input pair [1]. This score measures contributions to the change in the ground-truth-minus-adversarial logit margin. The paper does not establish what substantive information its dual-target construction contributes beyond attributing that margin directly, or how it advances existing class-aware neuron-importance approaches [3]. A matched comparison with direct margin attribution would clarify the methodological contribution.
  2. The contribution of NSA-derived weights to the reported robustness gains is not convincingly isolated. In Appendix Table 5, standard feature alignment (lambda = 0) already achieves 49.95% AutoAttack accuracy on ResNet-18 with DKL, compared with 50.19% using the proposed weighting (lambda = 1). This is a 0.24-point incremental improvement in that setting. Random or class-shuffled weights and simpler neuron scores should be compared under the identical recalibration loss to determine whether the specific NSA scores matter. The overall improvements in Table 1 are modest, and training evaluations are limited to CIFAR-10 and two CNN architectures.
  3. The neuron-masking experiments do not establish the claimed causal robustness mechanism. Section 4 uses ground-truth-class masks; Appendix A.3 states that adversarial examples for the visual analyses were generated against the original, unmasked network. Therefore, restored accuracy after masking does not demonstrate robustness to attacks adapted to the masked model or to a label-blind deployment setting. These experiments support NSA as a diagnostic ranking, but the claim that its identified neurons are the primary causal targets of attacks is stronger than the evidence warrants. Grad-CAM realignment alone does not bridge this gap.
  4. The sensitivity definition and attribution-to-training procedure need clarification. Section 3.1.2 defines the adversarial target as argmax f(x_adv). If an attacked input remains correctly classified, this target equals the ground-truth class and Equation (4) becomes zero for every neuron, despite potentially meaningful activation changes. It is unclear how such examples are handled. Furthermore, Appendix Algorithm 1 assumes precomputed class-wise masks without making sufficiently clear the source checkpoint, whether masks are refreshed as model parameters change, or the additional attribution/training cost. These choices are material to reproducibility and the plug-and-play claim.

[1] Shrikumar, A., Greenside, P., and Kundaje, A. “Learning Important Features Through Propagating Activation Differences.” International Conference on Machine Learning (ICML), 2017.

[2] Zhang, C., Liu, A., Liu, X., et al. “Interpreting and Improving Adversarial Robustness of Deep Neural Networks with Neuron Sensitivity.” IEEE Transactions on Image Processing, 30:1291–1304, 2021.

[3] Yan, H., Zhang, J., Niu, G., et al. “CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection.” International Conference on Machine Learning (ICML), 2021.

Questions for authors
  1. How are correctly classified adversarial inputs handled when the ground-truth and adversarial attribution targets coincide? From which model checkpoint are the class-wise masks computed, and are they updated during training?
  2. Under the same SANR objective, how do direct logit-margin attribution, random/class-shuffled weights, and activation-change-based sensitivity weights compare with NSA-derived weights?
  3. Are the Section 4 masking results intended solely as an interpretability diagnostic? Were attacks regenerated after masking, and what additional computational cost does NSA/SANR introduce?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8PMR

Dear Reviewer,

We sincerely thank you for your careful review, constructive critique, and insightful questions. Below, we address your concerns point by point.

Response to Weakness 1: Novelty of NSA vs. CIFS [3] Methods like CIFS operate on neuron importance (clean prediction contributions). However, an "important" neuron is not inherently "vulnerable." A strong feature's massive activation pushes it into the saturated regime of non-linear functions (Sigmoid/Tanh/Softmax). In CIFS, this neuron receives a high score because it anchors forward prediction. Yet, from an adversarial sensitivity perspective, its saturated state yields a near-zero local derivative, meaning a minute ϵ\epsilon-perturbation barely alters probability. Thus, important neurons are not equivalent to vulnerable ones. Furthermore, while direct margin attribution is possible, it obscures internal network dynamics. Our dual-target construction backpropagates directly from the output layer via two distinct causal paths, separately quantifying contributions to suppressing ground-truth confidence and promoting adversarial confidence, isolating hyper-sensitive neurons that direct margin differences cannot reveal.

Response to Weakness 2: Isolation of NSA Contribution and Scope We acknowledge that evaluating random or class-shuffled weight baselines under identical losses would rigorously isolate NSA utility. Due to the tight rebuttal window and GPU limits for retraining, we could not complete these control experiments in time. However, SANR is a proof-of-concept application to validate our analytical framework rather than chase SOTA benchmarks. The modest 0.24-point improvement over the baseline reflects an inherent characteristic of neuron-level adversarial defenses: standard AT already imposes an implicit penalty on vulnerable paths. Our method superimposes explicit, category-specific guidance, resulting in incremental refinement. Although ImageNet training is precluded by resource constraints during rebuttal, the NSA theoretical framework is inherently architecture-agnostic, and we will incorporate larger-scale evaluations in future work.

Response to Weakness 3: Causal Mechanism and Masking Evaluation We agree our evaluation—using adversarial examples generated against the unmasked network—does not constitute an evaluation against adaptive attacks tailored to the masked model. However, our objective is Diagnostic Ranking Validation: masking top-NSA-ranked sensitive neurons is diagnostic rather than claiming an impenetrable adaptive defense. Showing that masking restores accuracy demonstrates that these targeted neurons are primary pathways exploited by the adversary. Grad-CAM realignment further provides strong diagnostic evidence that disabling high-NSA neurons mitigates specific adversarial vulnerability. We have revised the manuscript to tone down over-emphatic causal claims.

Response to Weakness 4: Adversarial Success, Checkpoints, and Cost 1. Adversarial Success: Masks are computed using successfully attacked adversarial examples (where prediction diverges from ground truth). Rare instances where an attacked input remains correctly classified are naturally filtered out. 2. Checkpoints & Static Masks: Masks are extracted from the checkpoint achieving highest clean accuracy before adversarial training. They are not updated dynamically; they are pre-computed exactly once offline and remain static throughout Algorithm 1. 3. Computational Cost: Calculating masks requires two DeepLIFT passes per sample. While intensive, this is a one-time offline cost. By decoupling mask generation from training, Algorithm 1 incurs zero additional DeepLIFT overhead (taking a few hours on a single GPU for CIFAR-10). Changes: Incorporated into Appendix A.4.


Responses to Author Questions:

Q1: How are correctly classified adversarial inputs handled? Checkpoints and mask updates? Response: As detailed in Weakness 4, correctly classified inputs are filtered out, we use the checkpoint with highest clean accuracy, and masks remain completely static.

Q2: How do baseline weights compare with NSA? Response: Please refer to our discussions in Weakness 1 and 2 regarding the necessity of dual-target attribution over baseline approaches.

Q3: Are Section 4 masking results solely an interpretability diagnostic? Attacks regenerated? Cost? Response: Yes, Section 4 results are strictly an interpretability diagnostic rather than an adaptive defense, and attacks were not regenerated post-masking. Mask computation is a one-time offline process (taking a few hours on one GPU for CIFAR-10), introducing zero additional runtime overhead during training.

Thank you for your detailed response and clarifications. I appreciate that several concerns regarding the masking experiments and implementation details have been addressed. However, my main concerns about the novelty of NSA relative to direct margin attribution and the lack of controlled experiments isolating the contribution of NSA-derived weights remain unresolved. Therefore, I will maintain my original rating.

Reviewer_cinJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a neuron sensitivity measurement, building on DeepLIFT. The method attributes the logit change to neurons between a clean sample and an adversarial one, using both GT logit and adversarial logit. By manipulating the neuron activations and training to stablize them via an L1 loss, the network performance is improved marginally.

Critical strengths
  1. The evaluation of adversarial attack is thorough, including many kinds of them.
  2. The analysis of the failure case is appreciated.
  3. The paper is easy to follow.
Critical weaknesses
  1. The novelty and practicality of the paper is one of my major concerns. SNS [1] already computes class-specific sensitive neuron sets and show that different classes do not share them. It also shows that top sensitive neurons are also semantic in that suppressing them affects clean accuracy. The training loss design in SNS is also a L1 between clean and adversarial features. The new designs in SANR seem to be trivial in that it does not provide significant performance improvement. Most of adversarial improvements are around 0.5% on a small CIFAR10 dataset, which can largely fall in non-statistical-significant regions or due to training noises, e.g. performance change across different epochs. The proposed trichotomy in the paper, sensitive/marginal/robust, largely resembles that in [2].
  2. The experiment scope is limited, which is my second major concern. The experiments only involve small CNNs on a small CIFAR10 dataset. Even compared with SNS, there is no ImageNet demonstrations of SANR. Moreover, modern computer vision uses backbones such as ViT, WRN, ConvNeXt, which would be more valuable if their properties can be interpreted or their performance can be improved. The adversarial attack also only concerns L_inf case, where little L2 or other cases are considered.
  3. line 219 "This metric intuitively categorizes neurons", in this part, when S << 0, why couldn't it be the case that both C are larger than or smaller than 0? S is the difference between C, and it's linear in the chosen class (the S computed from two classes should agree with the one computed from margin of the logit according to the definition), so it seems to me that the C is not decidable from S alone. It could be me that do not understand the metric well, and I would appreciate some explanations here. This could potentially undermine the design validity.

[1] Interpreting and Improving Adversarial Robustness of Deep Neural Networks with Neuron Sensitivity. TIP 2020

[2] Interpreting adversarial examples by activation promotion and suppression.

Questions for authors

Please see Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cinJ

Dear Reviewer,

We sincerely thank you for your careful review and constructive suggestions. We address your concerns point by point below.

W1: Novelty, Practicality, and Relation to Prior Works [1, 2]

1. Novelty vs. SNS [1]: SNS evaluates neurons using absolute activation changes. This magnitude-only approach is insufficient: as Xu et al. (2019) noted, absolute changes encapsulate mixed components (suppressing the ground-truth label vs. promoting incorrect labels). Furthermore, intermediate activation variations do not necessarily propagate to the output layer. In contrast, NSA uses rigorous attribution, backpropagating directly from the output layer to mathematically decouple these mixed effects and accurately isolate neurons dictating misclassification.

2. SANR Performance Gains: While SANR borrows the training loss design from SNS, integrating NSA consistently yields baseline improvements. The modest gain (~0.5%) reflects a fundamental characteristic of neuron-level adversarial training: standard AT inherently imposes an implicit penalty on vulnerable neurons. NSA provides explicit, precise guidance on top of this, resulting in incremental leaps. Crucially, our primary contribution is the NSA analytical framework; SANR serves as a proof-of-concept application to empirically validate NSA, rather than a pursuit of state-of-the-art robustness metrics.

3. Trichotomy vs. [2]: Our trichotomy was inspired by [2]. However, [2] evaluates channel sensitivity via differences in intermediate activation regions Mk(x)M_k(x). Their methodology differs entirely: they do not distinguish whether changes ultimately promote or suppress the final prediction, and they ignore whether localized intermediate changes propagate to the output. Because NSA is grounded in attribution backpropagated from the final output, it accurately captures the causal chain of the error.

W2: Limited Experimental Scope (Datasets, Architectures, Attacks)

We completely agree that evaluating on larger datasets (CIFAR-100, Tiny-ImageNet, ImageNet), modern architectures (ViT, WRN, ConvNeXt), and diverse threat models (L2L_2 attacks) would significantly strengthen the paper's comprehensiveness.

Due to the strictly limited rebuttal window, extensive hyperparameter tuning, and substantial computational overhead for attribution analysis and AT at scale, we cannot report these large-scale experiments at this time. However, the NSA analytical framework is theoretically architecture-agnostic. The methodology of decoupling and attributing activation changes can be seamlessly generalized to modern vision backbones and diverse attack norms, which we are committed to scaling in our subsequent research.

W3: Clarification on Metric SS and Neuron Categorization

To address your concern regarding the case where Ck(true)(x)C^{(true)}_k(x) and Ck(adv)(x)C^{(adv)}_k(x) share the same sign (both >0>0 or both <0<0):

If a neuron's contributions to decreasing the ground-truth confidence and increasing the adversarial confidence are simultaneously large (or simultaneously small) with the same sign, computing Sk(c)(x)=Ck(true)(x)−Ck(adv)(x)S^{(c)}_k(x) = C^{(true)}_k(x) - C^{(adv)}_k(x) causes these two effects to mathematically cancel each other out, resulting in a value close to 0.

This specific scenario is exactly what we anticipated and captured in our definition of a Marginal Neuron:

"Marginal Neuron (Sk(c)(x)≈0S^{(c)}_k(x) \approx 0): The neuron is insensitive to both ground-truth and adversarial signals, or their opposing effects cancel each other out."

Therefore, for S≪0S \ll 0 to strictly hold (which categorizes a Sensitive Neuron), it mathematically necessitates a significant divergence between C(true)C^{(true)} and C(adv)C^{(adv)}. By filtering these canceling effects into the Marginal category, our metric rigorously preserves its design validity in isolating uniquely vulnerable neurons that distinctively favor the adversarial class over the true class.

[1] Interpreting and Improving Adversarial Robustness of Deep Neural Networks with Neuron Sensitivity. TIP 2020

[2] Interpreting adversarial examples by activation promotion and suppression.

Thank you for your rebuttal. I appreciate the effort in the further explanation. I understand that the proposed method update SNS in some ways, while it does not provide readers with new conclusions that are not in SNS, as mentioned above. The POC experiment using SANR does not bring enough gains to prove the effectiveness of the new neuron sensitivity analysis method. Moreover, the Metric S indeed mathematically necessitates a significant divergence between CtrueC^{true} and CadvC^{adv}, while it does not mean they should have different signs. Scaling remains to be a significant problem.

Overall, several of my major concerns remain, and I keep my score for now.

Reviewer_w6sjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper scores intermediate neurons with DeepLIFT, using the clean image as reference and the adversarial image as input. The score of a neuron is its contribution to the true-class logit change minus its contribution to the adversarial-class logit change, averaged over training images of each class. These class-wise scores are passed through tanh and used in an extra L1 loss (SANR) that pulls adversarial features toward a rescaled clean feature. The authors test the scores with neuron masking on CIFAR-10 and add SANR to AT, TRADES and DKL on ResNet-18 and VGG-16.

Critical strengths

Looking at the two directions separately (lowering the true class vs. raising the wrong class) makes sense, and the method is easy to add to existing adversarial training. I found the per-class masking results in the appendix interesting: the number of neurons that has to be masked varies a lot between classes and between the two networks. SANR improves all settings in Table 1, although by small margins.

Critical weaknesses
  1. The paper says that previous neuron-level methods are class-agnostic and presents NSA as a category-specific evaluation. I don't think this holds. Class-level analysis of neurons under attack has been done before, and some of the closest work is not discussed:

    • Kulkarni and Weng, "Interpretability-Guided Test-Time Adversarial Defense" (ECCV 2024), rank every neuron for every class and observe that successful attacks lower the activations of neurons important to the ground-truth class and raise those important to the predicted class. This is almost the same picture as in Section 3, and the paper is not cited.
    • CAS (Bai et al., 2021) learns one channel-importance vector per class and reweights channels with the ground-truth class during training. The related work only says that CAS suppresses low-contribution neurons.
    • The SNS paper (Zhang et al., 2021) also looks at which penultimate neurons drive the misclassification for each target class, and reports per-class correlation with neuron sensitivity.
  2. The gain from the sensitivity mask itself seems small. In Table 5, λ=0\lambda = 0, which is just feature alignment without NSA, already gets 49.95% AA, against 49.75% for DKL and 50.19% with λ=1\lambda = 1. So most of the improvement comes from the alignment loss and NSA adds around 0.2 points. AA gains in Table 1 are 0.4 to 0.6 points, with no variance over seeds. I would also like to see at least one larger dataset (CIFAR-100 or Tiny-ImageNet).

  3. γ\gamma in Eq. (6) is never specified, and the paper doesn't say how it should be chosen. β\beta and λ\lambda have ablations in A.6 but γ\gamma doesn't. It is also unclear how the "pre-computed sensitivity masks" in Algorithm 1 are obtained: on which model (standard or the one being trained), with which attack, whether they are updated, and at what cost (two DeepLIFT passes per sample).

  4. The writing overclaims in many places, for example "precisely characterizes", "deeply revealing", "direct causal link", "proves that rigid masking inevitably causes fatal semantic loss", "perfectly mirroring". An attribution score plus masking does not establish causality, and the Grad-CAM explanation in 4.3 is speculative. Parts of the introduction (e.g. the paragraph on DeepLIFT with a clean reference) repeat the abstract and read as if heavily polished by LLM.

Questions for authors

How much do NSA's top neurons overlap with IG-Defense's leave-one-out ranking and CAS's class weights on the same ResNet-18?

1 comment
AuthorsOpenReview ↗

Response to Reviewer w6sj

Dear Reviewer,

Thank you for the careful review. We address your concerns below.

W1: Prior Methods (IG-Defense, CAS, SNS)

Our core problem is that existing methods evaluate neurons by assessing sensitivity via activation changes, or measuring importance via contributions to clean samples. Neither captures the underlying association between activation changes and ultimate output variations. Crucially, this is not equivalent to claiming previous neuron-level methods are class-agnostic.

1. Importance-Based: "Interpretability-Guided Test-Time Adversarial Defense" (ECCV 2024) and CAS are importance-based. However, an "important" neuron doesn't mean attacking it easily causes model failure. Consider a salient feature (e.g., a dog's nose) whose massive activation pushes it into the saturation zone of activation functions (Sigmoid/Tanh/Softmax). It contributes significantly during forward propagation (yielding a high attribution score). Yet, its derivative is nearly zero. Adding a minute ϵ\epsilon perturbation trivially shifts the output probability. Thus, important neurons are not strictly equivalent to vulnerable ones.

2. Sensitivity-Based: SNS (Zhang et al., 2021) is sensitivity-based but evaluates absolute activation changes. This magnitude-only approach is insufficient because intermediate shifts encapsulate factors suppressing the ground-truth label, promoting incorrect labels, and mixed components [1](Xu et al., 2019). Our method bridges this gap by decoupling these distinct effects.

Changes: Added "Interpretability-Guided Test-Time Adversarial Defense" to Related Work.

W2: Performance Gain and Dataset Scale

1. Marginal gain: While feature alignment is a solid baseline, integrating NSA consistently yields further improvements. This modest gain reflects a fundamental characteristic: standard adversarial training inherently imposes an implicit penalty on vulnerable neurons. Our method provides explicit, precise, category-specific guidance on top of this implicit suppression, resulting in incremental rather than drastic leaps.

2. Core contribution: Our primary contribution lies in the NSA analytical framework. SANR serves as a proof-of-concept application to empirically validate NSA's effectiveness, rather than merely pursuing state-of-the-art robustness metrics.

3. Larger datasets: We agree that evaluating on larger datasets (CIFAR-100/Tiny-ImageNet) would strengthen the paper. Due to rebuttal time limits and computational overhead, we cannot report these now, but will in future work.

W3: Implementation Details for Mask Generation

We apologize for this oversight and revised the manuscript to clarify:

1. Model: Computed on a standard pre-trained model before training.

2. Attack: Generated via a 10-step PGD attack with max perturbation ϵ=8/255\epsilon=8/255 and step size α=2/255\alpha=2/255.

3. Updated? No. As "pre-computed" implies, masks are computed exactly once offline prior to Algorithm 1 and remain fixed.

4. Cost: Calculating masks requires two DeepLIFT passes per sample. While intensive, it is strictly a one-time offline cost. By decoupling mask generation from the training loop, actual robust optimization incurs zero extra DeepLIFT overhead. For CIFAR-10, this takes a few hours on one GPU.

Changes: Added these supplements to Appendix A.4.

W4: Overclaims in writing and causality

We agree our previous phrasing overstated conclusions; attribution methods with masking demonstrate strong correlation, not strict causality. We conducted a comprehensive revision:

1. Toning down overclaims: We eliminated all absolute and causal language. "Direct causal link" was revised to "clear connection" or "underlying association." "Precisely characterizes" was corrected to "quantitatively assesses."

2. Restructuring Introduction: We apologize for the repetitive structure and stylized language. We significantly rewrote the Introduction. Specifically, the DeepLIFT paragraph was condensed to focus purely on technical formulation, eliminating redundancy with the Abstract.

Question: Overlap with IG-Defense and CAS

Since our primary objective differs fundamentally, we didn't quantitatively compute the exact overlap of top-ranked neurons. We expect a partial overlap with significant divergences.

1. Overlap: A partial overlap exists because some neurons crucial for clean predictions also serve as highly effective adversarial targets.

2. Divergence: As clarified in W1, "importance" does not equate to "adversarial sensitivity." A top-ranked neuron in IG-Defense/CAS might reside in a saturation zone. While important to the clean decision boundary, its gradient is nearly zero, making it robust to minute ϵ\epsilon perturbations. Conversely, NSA identifies neurons possessing high gradient sensitivity that can be easily manipulated to flip predictions. [1] Interpreting adversarial examples by activation promotion and suppression, 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.