- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper evaluates the relationship between the sparsity of hidden representations and the adversarial robustness of neural networks. Specifically, the authors study whether activation-mask sparsity of a given sample, indicated by , can predict its distance to the decision boundary, . In other words, represents the minimum adversarial noise required to break the model on , and thus an indicator of the model robustness. To this purpose, the authors estimate this distance using DeepFool and a simplified CW-L2 attack. Once executed the analysis on datasets like MNIST and CIFAR10, and on a small CNN, the results suggest that sparsity alone is a poor predictor.
Critical strengths
The authors focus on the fundamental problem of understanding the relationship between adversarial robustness and hidden representations properties at the individual sample level. The overall goal is find the mechanism and properties that could lead to better robustness development in the future. The proposed analysis wants to go in this exploratory direction.
Critical weaknesses
Reliability of the adversarial distance estimates
My main concern is whether the attacks used in the paper provide sufficiently reliable estimates of to support the conclusions. The analysis relies on DeepFool and a simplified CWL2 attack. These methods provide candidate adversarial perturbations, but they do not guarantee that the closest decision boundary has been found. On the opposite they are known to be suboptimal estimator for minimum distance [x,y]. Therefore, the paper should distinguish more clearly between the minimum adversarial distance and the distance estimated by a particular attack.
[x] Attackbench: Evaluating gradient-based attacks for adversarial examples, AAAI 2025.
[y] Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses, CVPR 2019.
Given that the central contribution concerns correlations with that distance, if the approximation error varies across samples, it could weaken or alter the observed relationship between sparsity and robustness. I am not claiming that this explains the results, but the current analysis does not seem to rule out this possibility.
The authors should assess whether their conclusions remain consistent when using additional attacks and more extensive optimization (e.g., [y, k]). In particular, it would be useful to compare the perturbation norms obtained across attacks and repeat the correlation analysis using the smallest successful perturbation found for each sample. Without this check, I find it difficult to assess how much the conclusions depend on the chosen distance estimator.
[k] Fast minimum-norm adversarial attacks through adaptive norm constraints. NeurIps 2021.
Lack of a clear theoretical or intuitive justification
While reading the paper, I had the feeling that the motivation for expecting a relationship between activation sparsity and adversarial robustness is not sufficiently developed. For example, in the introduction, see lines 036–040, introduces this relationship without clearly explaining why sparsity should be informative about the distance to the decision boundary. What geometric mechanism would justify this expectation, and under which assumptions should it hold? An exploratory analysis can certainly be useful, however, a clearer hypothesis to guide the storytelling, presentation of the paper, and consequently also the experiments would help the reader interpret the negative result.
Limited empirical scope
The experimental analysis is confined to MNIST and CIFAR-10 and a single, small CNN architecture. While this setting is useful for an initial investigation, I do not think it is sufficient to support general conclusions about the relationship between representation sparsity and adversarial robustness. The relationship could depend on how representations are learned, and results from one small network do not establish whether the same behavior occurs in larger models or more sophisticated adversarially trained networks. The authors acknowledge these experimental limitations in the Appendix, which is strongly appreciated, but acknowledging them does not resolve the question of generality. I would find the conclusions more convincing with additional architectures and a more challenging dataset.
Questions for authors
Can the authors express their idea on wow sensitive are the reported correlations to the attack used to estimate ? Would the conclusions remain consistent when taking the smallest successful perturbation found across multiple attacks?
What is an intuitive or theoretical justification for expecting activation sparsity to predict adversarial robustness?
4 comments
Response (Cntd)
Weakness 3: limited empirical scope
We would like to correct the scope slightly: the paper covers four settings (MNIST, FashionMNIST, CIFAR-10, and a PGD adversarially trained CIFAR-10 model), five sparsity levels each, and, new in this revision, a depth study with plain ReLU CNNs of 2, 4 and 8 layers on MNIST and FashionMNIST (24 models; Section 5.9, Appendix E). The main pattern holds at every depth: attack-point D_M correlates 0.80–0.97 with R, while the δ-free measures stay at |r| ≤ 0.26. The adversarially trained models show the same pattern (the table above).
We do agree that this is not enough for claims about larger networks or harder datasets, so we added a residual architecture: CIFAR-style ResNet-20 (BatchNorm, 19 post-ReLU tensors pooled into the mask) on CIFAR-10 and CIFAR-100, at lambda in {0, 0.01}, one seed each (4 models), 500 examples per model, R = best of DeepFool and DDN (Section 5.10, Appendix H). There is no clean-input A(x) here (no hidden FC layer), so the delta-free control is fixed-radius sensitivity at 0.25, 0.5 and 1 times the median R. Results (Pearson r with R, per model):
| S(x) | attack-point D_M | delta-free D_M^eps | |
|---|---|---|---|
| CIFAR-10 | +0.41 to +0.44 | 0.89 to 0.91 | −0.38 to −0.36 |
| CIFAR-100 | +0.40 to +0.45 | 0.79 to 0.83 | −0.36 to −0.33 |
The coupling gap reproduces: the attack-point correlation is about 2.5 times the delta-free one (3 to 5 times in the small CNNs). We want to be clear about two differences, both stated in the paper. First, the delta-free residual is larger than in the small CNNs (|r| 0.33 to 0.38 versus at most 0.26). Second, S(x) is not a weak correlate within a ResNet-20 (+0.40 to +0.45 per model, versus −0.20 to +0.31 per lambda in the small CNNs); pooling the two lambdas hides this (+0.06, +0.08) because lambda = 0.01 raises S and lowers R. So the weak-sparsity finding is a finding about the small CNNs, and the revised abstract, introduction, discussion and conclusion say so. It is one seed and two lambdas, so we make no wider claim about residual networks. We have not tested attention architectures or non-ReLU activations. Our narrower claim is methodological: a statistic evaluated at an attack's output must be checked against a delta-free control before it is read as a robustness signal, and that check is cheap to run on any architecture.
We thank the reviewer again. The attack-sensitivity analysis was the most useful addition prompted by this review, and the conclusions held up under it.
Dear authors,
Thank you for the detailed answer and further experiments. The strongest attacks and the comparison based on the smallest successful perturbation substantially address my concern about distance estimates.
What remains less convincing is the connection between activation sparsity and robustness. The local linear argument explains the role of the margin norm and the gradient, but does not establish how sparsity affects these quantities. Fewer active units do not necessarily imply a smaller gradient norm, and any advantage also depends on how the margin changes. While I appreciate the rebuttal and the additional explanation, I remain concerned that the motivation for linking activation sparsity to robustness is not sufficiently supported. The argument offers a plausible intuition, but relies on assumptions that have neither been theoretically justified for the studied models nor directly tested in the experiments.
Furthermore, from an empirical point of view, ResNet results also deserve more attention. Sparsity correlates moderately with robustness within each model, and this largely disappears under pooling regularization settings. We can thus conclude that the conclusion depends substantially on the architecture and how the results are aggregated. For non-ReLU networks, for example, all the stated assumption never holds.
Finally, evaluation on adversarially robust models remains weak in terms of number of models tested and their robustness level comapred to the state of the art. The authors provide a weak PGD-trained CIFAR-10 CNN, but that doesn't determine whether the results extend to stronger, more established robust models (that can be downloaded for free from the RobustBench repository). The additional ResNet experiments expand the architectural scope (while remaining on small and relu-based models), but no adversarially trained ResNet results are reported.
Reply
Thank you for confirming that the attack-sensitivity analysis resolves the distance-estimation concern.
On your remaining points:
-
Sparsity-robustness link. You are right that the linearization yields a hypothesis, not a mechanism: fewer active units do not imply a smaller gradient norm, and the margin can move too. Our paper does not claim the link holds. It reports that it largely does not in the small CNNs. To make the negative result diagnostic, we now test each assumption per example: (A1) corr(S, margin) = [x]; (A2) actual gradient norm vs. the norm predicted under proportional path deletion = [x]; (A3) fraction of examples with no mask switch before the boundary, and corr(S, R) on that subset = [x]. We will add these to the camera-ready.
-
Architecture and pooling. We agree that the within-model relation depends on architecture (weak in small CNNs, moderate in ResNet-20). The pooled ResNet result is a between-model confound: λ = 0.01 raises S and lowers R, masking the within-model correlation. We see this as supporting our methodological point that sparsity-robustness correlations must be evaluated within model. The linearization argument applies only to piecewise-linear activations, so we have restricted our claims to ReLU-family networks.
-
Robust models. We agree a weak PGD-trained CNN is insufficient. We ran the RobustBench CIFAR-10 L2 models, with R = best of DeepFool and DDN on 500 examples each. These are single checkpoints, so they test whether the within-model relation and coupling gap persist at higher robustness, not how sparsity affects robustness across training.
The revised paper claims only that (a) in the studied ReLU CNNs, per-example sparsity is a weak predictor of attack-estimated distance, (b) statistics evaluated at an attack's output are coupled to R by construction and need a delta-free control, and (c) pooling across regularization settings can mask or create correlations.
Thank you again.
Response
We thank the reviewer for a precise review. The first concern, how much the conclusions depend on the attack used to estimate R(x), is one we could test directly, and we have. The result is below with the numbers.
Weakness 1 / Q1: reliability of the distance estimate; do conclusions hold with stronger attacks and the smallest successful perturbation?
We agree that DeepFool gives an upper bound on the minimum adversarial distance, not the distance itself. We now run three stronger minimum-norm L2 attacks on all 20 checkpoints, on the same 500 examples per model as before: DDN (Rony et al. 2019; 1000 steps), FMN-L2 (Pintor et al. 2021; 1000 steps) and full Carlini–Wagner L2 with 9 binary-search steps and 1000 steps (Foolbox 3.3.4 reference implementations, which replace the cheaper 4-value-grid CW of the original cross-check). For each example we also form R_best, the smallest successful perturbation across all four attacks, and repeat the whole correlation analysis with it (pooled n = 2500 per dataset; Section 5.8, Table 6 and Appendix C). All attacks succeed on 100% of examples.
Is DeepFool suboptimal? Yes. DeepFool's R is larger than the best-of-4 R by a median factor of 1.39 (MNIST), 1.37 (FashionMNIST), 1.08 (CIFAR-10) and 1.10 (CIFAR-10-AT). The best attack is mostly DDN or CW-L2. (FMN-L2 with our settings is usually no better than DeepFool and is never the minimum on any dataset; we read this as under-tuning, and no conclusion depends on it.)
Does the error change the correlations? The estimation error is largely a per-model, near-uniform rescaling: corr(R_DeepFool, R_best) = 0.95 / 0.98 / 0.99 / 0.97 (MNIST / FashionMNIST / CIFAR-10 / CIFAR-10-AT). Pooled Pearson correlation with R (Table 6):
Across all four attacks and R_best, r(S, R) stays in [−0.19, +0.10], and the δ-free measures stay at |r| ≤ 0.24. The attack-point D_M correlation stays at 0.74–0.94 under every attack, so the coupling result does not depend on which attack is used, which is expected, since the coupling comes from where the measurement is taken.
Two things the reviewer should know:
We have also revised the text to say "attack-estimated adversarial distance R̂(x), an upper bound on the minimum" where it said "adversarial distance", and R_best is now reported alongside DeepFool throughout. best-of-4 is still not a certified minimum (we make no claim of optimality), and we cite AttackBench for the caveat.
Weakness 2 / Q2: justification for expecting sparsity to predict robustness
This is a fair criticism of the original introduction, which asserted the link. The revised introduction (new paragraph "Why expect a relationship?") gives the mechanism. Within the activation region of x, a ReLU network is affine, f(x′) = J_M x′ + b_M, with J_M = W_L D_{L−1} W_{L−1} ⋯ D_1 W_1 and D_l = diag(M_l). The first-order distance to the boundary is R_lin(x) = m(x) / ‖J_Mᵀ(e_y − e_j)‖₂, with margin m(x) = f_y − f_j (this is DeepFool's first step). Zeroing a unit removes the paths through it from J_M, so if deleted paths carry gradient mass in proportion to their number, sparser masks give a smaller gradient norm and a larger R_lin.
This needs three assumptions, and the paper states them as such:
Each can fail, so whether S(x) predicts R(x) within a model is open, and a null result tells us which assumption fails. That is how we now read our results: the weak S–R correlation says the assumptions do not hold well enough to give a usable per-example signal, and assumption 3 is what the stability analysis in Sections 5.3–5.8 examines. We do not claim to know which assumption breaks.