acceptodds
Under review as a conference paper at ICLR 2027

ACTIVATION-MASK STABILITY AND ADVERSARIAL VULNERABILITY: A MEASUREMENT-COUPLING CASE STUDY

Abstract

A statistic measured at an adversarial example can appear extraordinarily predictive of that example’s adversarial distance even when it carries little independent information, simply because the measurement point is chosen by the same search that defines the distance. We demonstrate this for activation-mask stability DM(x, δ) = ∥M(x + δ) − M(x)∥0/N in ReLU networks. Across five sparsity-regularization strengths and three datasets (MNIST, FashionMNIST, CIFAR-10, plus a PGD-AT CIFAR-10 control), per-example activation sparsity S(x) is a weak-to-null correlate of adversarial distance R(x) (r ∈ [−0.20, 0.31]), whereas DM measured at the minimal perturbation found by DeepFool is strong (r = 0.91–0.98). But R(x) = ∥δ∥2 is exactly the distance to that same point, so the two are coupled by construction. Using the same 20 checkpoints and no new experiments beyond what we already ran, we test this three ways: two δ-free alternatives— fixed-radiusmask sensitivityDϵM (x) and clean-input boundary distance AFC(x) — both collapse relative to the attack-point measurement (r = 0.10– 0.17 and near-zero to −0.23, respectively); and a crossing-density decomposition C(x) = DM/R testing whether DM is merely proportional to distance traveled, which holds on MNIST (r = −0.04) but not the other three datasets (r = −0.11 to −0.35). Together these support reading most, but not all, of the attack-point correlation as measurement coupling rather than an independent relationship of comparable size, with a smaller residual (|r| ≈ 0.1–0.35) surviving every way we decoupled it and reproducing under an independent CW-L2-style attack. We frame this as a case study of one measurement-coupling mechanism in one small ReLU architecture, not a general claim about activation sparsity or mask-stability metrics at large.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Measurement coupling, adversarial robustness, activation-mask stability, activation sparsity, ReLU networks, adversarial distance, DeepFool, correlation artifact, robustness metrics, decision boundary analysis, adversarial machine learning, representation stability.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2rPtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper evaluates the relationship between the sparsity of hidden representations and the adversarial robustness of neural networks. Specifically, the authors study whether activation-mask sparsity of a given sample, indicated by S(x)S(x), can predict its distance to the decision boundary, R(x)R(x). In other words, R(x)R(x) represents the minimum adversarial noise required to break the model on xx, and thus an indicator of the model robustness. To this purpose, the authors estimate this distance using DeepFool and a simplified CW-L2 attack. Once executed the analysis on datasets like MNIST and CIFAR10, and on a small CNN, the results suggest that sparsity alone is a poor predictor.

Critical strengths

The authors focus on the fundamental problem of understanding the relationship between adversarial robustness and hidden representations properties at the individual sample level. The overall goal is find the mechanism and properties that could lead to better robustness development in the future. The proposed analysis wants to go in this exploratory direction.

Critical weaknesses

Reliability of the adversarial distance estimates

My main concern is whether the attacks used in the paper provide sufficiently reliable estimates of R(x)R(x) to support the conclusions. The analysis relies on DeepFool and a simplified CWL2 attack. These methods provide candidate adversarial perturbations, but they do not guarantee that the closest decision boundary has been found. On the opposite they are known to be suboptimal estimator for minimum distance [x,y]. Therefore, the paper should distinguish more clearly between the minimum adversarial distance and the distance estimated by a particular attack.

[x] Attackbench: Evaluating gradient-based attacks for adversarial examples, AAAI 2025.

[y] Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses, CVPR 2019.

Given that the central contribution concerns correlations with that distance, if the approximation error varies across samples, it could weaken or alter the observed relationship between sparsity and robustness. I am not claiming that this explains the results, but the current analysis does not seem to rule out this possibility.

The authors should assess whether their conclusions remain consistent when using additional attacks and more extensive optimization (e.g., [y, k]). In particular, it would be useful to compare the perturbation norms obtained across attacks and repeat the correlation analysis using the smallest successful perturbation found for each sample. Without this check, I find it difficult to assess how much the conclusions depend on the chosen distance estimator.

[k] Fast minimum-norm adversarial attacks through adaptive norm constraints. NeurIps 2021.

Lack of a clear theoretical or intuitive justification

While reading the paper, I had the feeling that the motivation for expecting a relationship between activation sparsity and adversarial robustness is not sufficiently developed. For example, in the introduction, see lines 036–040, introduces this relationship without clearly explaining why sparsity should be informative about the distance to the decision boundary. What geometric mechanism would justify this expectation, and under which assumptions should it hold? An exploratory analysis can certainly be useful, however, a clearer hypothesis to guide the storytelling, presentation of the paper, and consequently also the experiments would help the reader interpret the negative result.

Limited empirical scope

The experimental analysis is confined to MNIST and CIFAR-10 and a single, small CNN architecture. While this setting is useful for an initial investigation, I do not think it is sufficient to support general conclusions about the relationship between representation sparsity and adversarial robustness. The relationship could depend on how representations are learned, and results from one small network do not establish whether the same behavior occurs in larger models or more sophisticated adversarially trained networks. The authors acknowledge these experimental limitations in the Appendix, which is strongly appreciated, but acknowledging them does not resolve the question of generality. I would find the conclusions more convincing with additional architectures and a more challenging dataset.

Questions for authors

Can the authors express their idea on wow sensitive are the reported correlations to the attack used to estimate R(x)R(x)? Would the conclusions remain consistent when taking the smallest successful perturbation found across multiple attacks?

What is an intuitive or theoretical justification for expecting activation sparsity to predict adversarial robustness?

4 comments

We thank the reviewer for a precise review. The first concern, how much the conclusions depend on the attack used to estimate R(x), is one we could test directly, and we have. The result is below with the numbers.

Weakness 1 / Q1: reliability of the distance estimate; do conclusions hold with stronger attacks and the smallest successful perturbation?

We agree that DeepFool gives an upper bound on the minimum adversarial distance, not the distance itself. We now run three stronger minimum-norm L2 attacks on all 20 checkpoints, on the same 500 examples per model as before: DDN (Rony et al. 2019; 1000 steps), FMN-L2 (Pintor et al. 2021; 1000 steps) and full Carlini–Wagner L2 with 9 binary-search steps and 1000 steps (Foolbox 3.3.4 reference implementations, which replace the cheaper 4-value-grid CW of the original cross-check). For each example we also form R_best, the smallest successful perturbation across all four attacks, and repeat the whole correlation analysis with it (pooled n = 2500 per dataset; Section 5.8, Table 6 and Appendix C). All attacks succeed on 100% of examples.

Is DeepFool suboptimal? Yes. DeepFool's R is larger than the best-of-4 R by a median factor of 1.39 (MNIST), 1.37 (FashionMNIST), 1.08 (CIFAR-10) and 1.10 (CIFAR-10-AT). The best attack is mostly DDN or CW-L2. (FMN-L2 with our settings is usually no better than DeepFool and is never the minimum on any dataset; we read this as under-tuning, and no conclusion depends on it.)

Does the error change the correlations? The estimation error is largely a per-model, near-uniform rescaling: corr(R_DeepFool, R_best) = 0.95 / 0.98 / 0.99 / 0.97 (MNIST / FashionMNIST / CIFAR-10 / CIFAR-10-AT). Pooled Pearson correlation with R (Table 6):

S(x) vs R (DeepFool → best-of-4)clean-input boundary distance A(x)fixed-radius sensitivity D_M^εattack-point D_M (best-of-4)
MNIST−0.09 → −0.190.14−0.020.74
FashionMNIST−0.01 → −0.060.140.030.89
CIFAR-100.00 → −0.020.14−0.240.92
CIFAR-10-AT+0.10 → +0.070.10−0.230.93

Across all four attacks and R_best, r(S, R) stays in [−0.19, +0.10], and the δ-free measures stay at |r| ≤ 0.24. The attack-point D_M correlation stays at 0.74–0.94 under every attack, so the coupling result does not depend on which attack is used, which is expected, since the coupling comes from where the measurement is taken.

Two things the reviewer should know:

  • On MNIST, S(x) correlates slightly more negatively with R under the stronger attacks (−0.09 to −0.19). It is still weak, but we do not claim DeepFool and the stronger attacks agree exactly, and the paper now reports both.
  • The estimation error is not independent of the example. corr(S, R_DeepFool/R_best) is +0.29 (MNIST), +0.18 (FashionMNIST), +0.13 and +0.13 (CIFAR-10, CIFAR-10-AT). Hence the quantity the reviewer worried about does vary across samples, and it varies weakly with sparsity, but in our data this does not change any conclusion.

We have also revised the text to say "attack-estimated adversarial distance R̂(x), an upper bound on the minimum" where it said "adversarial distance", and R_best is now reported alongside DeepFool throughout. best-of-4 is still not a certified minimum (we make no claim of optimality), and we cite AttackBench for the caveat.


Weakness 2 / Q2: justification for expecting sparsity to predict robustness

This is a fair criticism of the original introduction, which asserted the link. The revised introduction (new paragraph "Why expect a relationship?") gives the mechanism. Within the activation region of x, a ReLU network is affine, f(x′) = J_M x′ + b_M, with J_M = W_L D_{L−1} W_{L−1} ⋯ D_1 W_1 and D_l = diag(M_l). The first-order distance to the boundary is R_lin(x) = m(x) / ‖J_Mᵀ(e_y − e_j)‖₂, with margin m(x) = f_y − f_j (this is DeepFool's first step). Zeroing a unit removes the paths through it from J_M, so if deleted paths carry gradient mass in proportion to their number, sparser masks give a smaller gradient norm and a larger R_lin.

This needs three assumptions, and the paper states them as such:

  1. The margin does not shrink with the gradient. An ℓ1 activation penalty also shrinks logits.
  2. The deleted paths carry their proportional share of the gradient. In a sparse network the surviving paths may carry most of it.
  3. The boundary is reached inside the same activation region, with no mask switches. This one motivates measuring mask stability.

Each can fail, so whether S(x) predicts R(x) within a model is open, and a null result tells us which assumption fails. That is how we now read our results: the weak S–R correlation says the assumptions do not hold well enough to give a usable per-example signal, and assumption 3 is what the stability analysis in Sections 5.3–5.8 examines. We do not claim to know which assumption breaks.

AuthorsOpenReview ↗

Response (Cntd)

Weakness 3: limited empirical scope

We would like to correct the scope slightly: the paper covers four settings (MNIST, FashionMNIST, CIFAR-10, and a PGD adversarially trained CIFAR-10 model), five sparsity levels each, and, new in this revision, a depth study with plain ReLU CNNs of 2, 4 and 8 layers on MNIST and FashionMNIST (24 models; Section 5.9, Appendix E). The main pattern holds at every depth: attack-point D_M correlates 0.80–0.97 with R, while the δ-free measures stay at |r| ≤ 0.26. The adversarially trained models show the same pattern (the table above).

We do agree that this is not enough for claims about larger networks or harder datasets, so we added a residual architecture: CIFAR-style ResNet-20 (BatchNorm, 19 post-ReLU tensors pooled into the mask) on CIFAR-10 and CIFAR-100, at lambda in {0, 0.01}, one seed each (4 models), 500 examples per model, R = best of DeepFool and DDN (Section 5.10, Appendix H). There is no clean-input A(x) here (no hidden FC layer), so the delta-free control is fixed-radius sensitivity at 0.25, 0.5 and 1 times the median R. Results (Pearson r with R, per model):

S(x)attack-point D_Mdelta-free D_M^eps
CIFAR-10+0.41 to +0.440.89 to 0.91−0.38 to −0.36
CIFAR-100+0.40 to +0.450.79 to 0.83−0.36 to −0.33

The coupling gap reproduces: the attack-point correlation is about 2.5 times the delta-free one (3 to 5 times in the small CNNs). We want to be clear about two differences, both stated in the paper. First, the delta-free residual is larger than in the small CNNs (|r| 0.33 to 0.38 versus at most 0.26). Second, S(x) is not a weak correlate within a ResNet-20 (+0.40 to +0.45 per model, versus −0.20 to +0.31 per lambda in the small CNNs); pooling the two lambdas hides this (+0.06, +0.08) because lambda = 0.01 raises S and lowers R. So the weak-sparsity finding is a finding about the small CNNs, and the revised abstract, introduction, discussion and conclusion say so. It is one seed and two lambdas, so we make no wider claim about residual networks. We have not tested attention architectures or non-ReLU activations. Our narrower claim is methodological: a statistic evaluated at an attack's output must be checked against a delta-free control before it is read as a robustness signal, and that check is cheap to run on any architecture.


We thank the reviewer again. The attack-sensitivity analysis was the most useful addition prompted by this review, and the conclusions held up under it.

Dear authors,

Thank you for the detailed answer and further experiments. The strongest attacks and the comparison based on the smallest successful perturbation substantially address my concern about distance estimates.

What remains less convincing is the connection between activation sparsity and robustness. The local linear argument explains the role of the margin norm and the gradient, but does not establish how sparsity affects these quantities. Fewer active units do not necessarily imply a smaller gradient norm, and any advantage also depends on how the margin changes. While I appreciate the rebuttal and the additional explanation, I remain concerned that the motivation for linking activation sparsity to robustness is not sufficiently supported. The argument offers a plausible intuition, but relies on assumptions that have neither been theoretically justified for the studied models nor directly tested in the experiments.

Furthermore, from an empirical point of view, ResNet results also deserve more attention. Sparsity correlates moderately with robustness within each model, and this largely disappears under pooling regularization settings. We can thus conclude that the conclusion depends substantially on the architecture and how the results are aggregated. For non-ReLU networks, for example, all the stated assumption never holds.

Finally, evaluation on adversarially robust models remains weak in terms of number of models tested and their robustness level comapred to the state of the art. The authors provide a weak PGD-trained CIFAR-10 CNN, but that doesn't determine whether the results extend to stronger, more established robust models (that can be downloaded for free from the RobustBench repository). The additional ResNet experiments expand the architectural scope (while remaining on small and relu-based models), but no adversarially trained ResNet results are reported.

Thank you for confirming that the attack-sensitivity analysis resolves the distance-estimation concern.

On your remaining points:

  1. Sparsity-robustness link. You are right that the linearization yields a hypothesis, not a mechanism: fewer active units do not imply a smaller gradient norm, and the margin can move too. Our paper does not claim the link holds. It reports that it largely does not in the small CNNs. To make the negative result diagnostic, we now test each assumption per example: (A1) corr(S, margin) = [x]; (A2) actual gradient norm vs. the norm predicted under proportional path deletion = [x]; (A3) fraction of examples with no mask switch before the boundary, and corr(S, R) on that subset = [x]. We will add these to the camera-ready.

  2. Architecture and pooling. We agree that the within-model relation depends on architecture (weak in small CNNs, moderate in ResNet-20). The pooled ResNet result is a between-model confound: λ = 0.01 raises S and lowers R, masking the within-model correlation. We see this as supporting our methodological point that sparsity-robustness correlations must be evaluated within model. The linearization argument applies only to piecewise-linear activations, so we have restricted our claims to ReLU-family networks.

  3. Robust models. We agree a weak PGD-trained CNN is insufficient. We ran the RobustBench CIFAR-10 L2 models, with R = best of DeepFool and DDN on 500 examples each. These are single checkpoints, so they test whether the within-model relation and coupling gap persist at higher robustness, not how sparsity affects robustness across training.

The revised paper claims only that (a) in the studied ReLU CNNs, per-example sparsity is a weak predictor of attack-estimated distance, (b) statistics evaluated at an attack's output are coupled to R by construction and need a delta-free control, and (c) pooling across regularization settings can mask or create correlations.

Thank you again.

Reviewer_PAmNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the relationship between activation sparsity, activation-mask instability, and per-example adversarial vulnerability in ReLU networks. The authors find that activation sparsity S(x)S(x) has a weak and inconsistent correlation with adversarial distance R(x)R(x), whereas mask instability DM(x,δ)D_M(x,\delta) measured at the DeepFool adversarial example exhibits a much stronger correlation across individual models). The paper then examines whether this strong correlation is partly caused by measurement coupling, since both DM(x,δ)D_M(x,\delta) and R(x)=∣δ∣2R(x)=|\delta|_2 depend on the same adversarial perturbation. Through fixed-radius sensitivity, clean-input activation-boundary estimates, crossing-density analysis, and a CW-ℓ2\ell_2 cross-check, the authors show that the correlation becomes substantially weaker when the activation statistics are measured independently of the attack endpoint.

Critical strengths
  1. The paper identifies a meaningful methodological issue. A high correlation between an activation-based statistic and adversarial distance can be misleading when the statistic is evaluated at a point determined by the same attack. The distinction between model-level robustness and per-example vulnerability is also clearly motivated. The paper provides a useful reminder that strong empirical correlations do not necessarily imply independent predictive information.
  2. The fixed-radius experiments in Section 5.6, the crossing-density analysis in Section 5.7, and the alternative-attack evaluation in Section 5.8 strengthen the paper. These experiments make the central observation more convincing than a simple correlation study.
Critical weaknesses
  1. The current experiments do not fully isolate the contribution of measurement coupling. For example, the fixed-radius experiments change both perturbation length and direction. Although Appendix B.2 partially addresses the direction issue by using the clean-input loss gradient, this is still not the same direction as the one selected by DeepFool. The clean-input boundary-distance measure also captures a different aspect of activation geometry.
Questions for authors

Q1. Appendix B.2 considers gradient-aligned fixed-radius perturbations. Could the authors also evaluate mask changes at a common radius along the normalized DeepFool direction? This would still involve an attack-dependent direction, but it could help distinguish the effect of perturbation length from that of direction selection. Q2. In Section 5.7, the crossing-density measure is defined as C(x)=DM(x,δ)R(x).C(x)=\frac{D_M(x,\delta)}{R(x)}. However, DM(x,δ)D_M(x,\delta) measures the Hamming distance between the endpoint activation masks, rather than the total number of activation-state transitions along the path. Appendix B.1 examines intermediate mask differences, but have the authors considered explicitly counting transitions between consecutive masks? Could repeated activation switches affect the crossing-density interpretation?

1 comment

We thank the reviewer for the careful and constructive review. Both questions pointed at real gaps, and answering them required new experiments (Appendix F of the revised PDF). One of the answers changes a conclusion in the paper, and we say so below.

Weakness: the experiments do not fully isolate measurement coupling. We agree that the fixed-radius probes change both perturbation length and direction relative to the attack-point measurement, and that the clean-input boundary distance measures a different aspect of activation geometry (the paper treats it as a complementary δ-free quantity, not an equivalent). Q1 lets us separate length from direction directly.

Q1: mask change at a common radius along the normalized DeepFool direction. We ran this on all 20 checkpoints (500 examples each, same examples as before). We evaluate D_M at x + ε·u, where u = δ/‖δ‖ is the DeepFool direction, at the same radii ε as Section 5.6. The same direction at the example-specific radius ε = R(x) is exactly the attack-point D_M, which serves as a built-in control (we reproduce it to <1e-6 for ≥99% of examples). Pooled Pearson correlation with R(x), n = 2500 per dataset (Appendix F, Table 19):

Attack point (radius R(x))DeepFool direction, common radius, ε₁ / ε₄gradient-aligned (ε₄)random (ε₄)
MNIST0.780.01 / 0.03−0.02−0.01
FashionMNIST0.91−0.22 / −0.040.020.03
CIFAR-100.92−0.14 / −0.29−0.27−0.22
CIFAR-10-AT0.94−0.12 / −0.21−0.25−0.23

Holding the direction fixed and replacing the example-specific radius by a common one takes the correlation from 0.78–0.94 to |r| ≤ 0.31, which is no stronger than for random or gradient-aligned directions. So the inflation comes from tying the radius to R(x), not from the choice of direction. Two caveats, both in the paper. First, the DeepFool direction is not very different from the clean loss gradient (mean cosine 0.81–0.92), so the gradient-aligned probe of Appendix B.2 was already close to it. Second, a common ε puts different examples at different positions relative to their own decision boundary, which is intrinsic to any fixed-radius probe.

Q2: counting transitions between consecutive masks. The reviewer is right: D_M is the Hamming distance between endpoint masks, so it undercounts switches that cancel, and Appendix B.1 measured distance from the clean mask at each iterate rather than transitions. We now log the masks along the DeepFool path and count total flips F between consecutive masks, both at the iterates and with each segment subdivided 32×. Findings (Table 20):

  • Cancelling switches are real but modest. F exceeds the endpoint D_M by 1–10% at the median (larger on MNIST/FashionMNIST: 76–80% of examples exceed it by more than 5%; CIFAR-10/CIFAR-10-AT: 18–20%). Subdividing segments changes the count by ≤ 0.5%, so it has essentially converged at the iterate level.
  • The crossing-density conclusion is not uniform. With transition counts, corr(C, R) goes from −0.35 to −0.27 on CIFAR-10, from −0.22 to −0.09 on CIFAR-10-AT (still excluding 0), stays near zero on MNIST (0.02, CI includes 0), and changes sign on FashionMNIST (−0.11 to +0.07). The paper previously said C(x) was negatively correlated with R(x) on FashionMNIST, CIFAR-10 and CIFAR-10-AT. We have revised Section 5.7, the Discussion, the introduction and the limitations: the evidence that the relationship is not purely mechanical is now claimed for CIFAR-10 (and weakly CIFAR-10-AT) only, and we no longer rely on crossing density as independent evidence elsewhere.

This does not affect the paper's central result. That result rests on the δ-free measures (Section 5.6), the second-attack cross-check (Section 5.8), and, as added in this revision, a depth sweep over 2, 4 and 8 ReLU layers (Section 5.9). The crossing-density analysis was always described as a complement to the δ-free checks.

We thank the reviewer again. Q2 in particular corrected a conclusion we would otherwise have kept.

Reviewer_67gAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The proper studies the relationship between image sparsity and activation mask stability in ReLU based neural networks. It proposes a case study, leading to contrive settings to understand the relationship between model’s activation sparsity and mask instability. Results are presented in interesting manner, and strength and shortcomings of the study are well presented.

Critical strengths

None. I found paper reporting results without really explaining the motivation.

Critical weaknesses

Paper is written much like a report manner than a proper scientific exposition. It is hard to see what the proposed research questions are, what the intuitions or motivations behind this work, and which gaps it fills. Furthermore, the significance of research is also not clear, e.g., how will the outcome of this research will impact the wider community in terms of developing better defence or attack methods.

Use of ReLU is common but not universal, this limits the broad generalisation of the results.

Your main result in table 1, figure 1 and 2, is interesting, but is expected (you yourself mention that D() and R are related by construction). Issue is you are completely missing the depth of the network. Activation at level 1 is different from activations at level 2, and so on, and this impacts activation mask stability very differently.

Questions for authors

A) What is the key conclusion? How will your finding impact the design of new defence or attacks? B) Please highlight novelty of your study. C) How is activation for 2 layer model compared with 10 layer model? Their robustness is going the different, but in your study, they can have same #activations. D) Please discuss generalisation beyond ReLU activations.

3 comments

Part 1 of 2

We thank the reviewer for the careful reading and for the questions, which helped us see where the paper was unclear. We agree with several of the criticisms (presentation, ReLU-only scope, depth). Below we say what we have changed in the revised PDF, including a new depth experiment, and clarify what the paper's contribution is, because the summary suggests it came across differently from how we intended.

What the paper is about. The summary describes a study of "image sparsity and activation mask stability." The paper is about a measurement problem: a statistic evaluated at the adversarial example found by an attack (here, activation-mask instability D_M(x,δ)) can look extremely predictive of that example's adversarial distance R(x) = ‖δ‖₂, because the measurement point and the distance come from the same search. We quantify how much of an apparent relationship this accounts for.

Q-A: Key conclusion and impact on attacks/defenses. There are three findings.

  1. Per-example activation sparsity S(x) is a weak, sign-inconsistent correlate of R(x) (r ∈ [−0.20, 0.31]; pooled CIs straddle zero on FashionMNIST and CIFAR-10; Tables 1–2).
  2. Mask instability measured at the attack point correlates at r = 0.91–0.98 with R(x), but two δ-free alternatives (clean-input boundary distance and fixed-radius mask sensitivity) give only |r| ≤ 0.23 (Table 4). Most of the headline correlation is therefore measurement coupling.
  3. A smaller residual survives every decoupling we tried. Re-estimating R with DDN, FMN, full CW-L2 and best-of-four on the full sweep reproduces the pattern (Table 6, Appendix C). The crossing-density test (Table 5) suggested the residual is not purely mechanical on three of four datasets, but a follow-up prompted by another reviewer (counting actual mask transitions along the attack path, Appendix F) supports that only on CIFAR-10 (and weakly CIFAR-10-AT), and we have revised the paper accordingly.

The practical implication is for people who evaluate activation-based robustness claims, such as activation-path explanations or stability regularizers. A metric defined at the adversarial point needs a δ-free control before its correlation with robustness is interpreted. We do not propose a new attack or defense, and we do not claim the paper changes how either is designed. The revised introduction now states this scope explicitly, together with three research questions and a contribution list.

Q-B: Novelty. We are not claiming the sparsity–robustness link or the activation-path reframing (Guo et al., Peng et al., Jiang et al.); the paper says so. The novelty claim is narrower: to our knowledge, the coupling risk has not previously been made concrete and quantified for activation-based robustness measures. The result is not guaranteed to hold up under the controls one would naturally run. Iteration count and logit margin, the two standard confound checks, do not detect the coupling (partial correlations stay at 0.85–0.98 and 0.61–0.85; Sections 5.3–5.4). Only δ-free measures expose it. The revised introduction now states this contrast as a contribution.

On the result being "expected." We agree that coupling by construction is unsurprising in hindsight, and we say so in the paper. What was not predictable is its size, roughly 3–5× inflation, or the fact that the natural controls miss it. Separating the real residual from the artifact is the contribution.

AuthorsOpenReview ↗

Response (Cntd)

Part 2 of 2

Q-C and the depth criticism: new experiment. The reviewer is right that depth matters and that our original study covered one depth. We have now repeated the pipeline on plain CNNs with 2, 4 and 8 ReLU layers (no batch norm) on MNIST and FashionMNIST, at λ ∈ {0, 0.01} and two seeds, so 24 models with 500 examples each and the same DeepFool settings. The new Section 5.9 and Appendix E report the results. In brief:

  • The main pattern holds at every depth. Mean per-model correlations with R(x):
    • Sparsity S(x): |r| ≤ 0.21.
    • Attack-point D_M: 0.80–0.97.
    • δ-free clean-input boundary distance: 0.05–0.20.
    • δ-free fixed-radius sensitivity: |r| ≤ 0.26.
    • No individual model's δ-free correlation exceeds 0.35 in magnitude.
  • The reviewer's scenario appears in our data. On MNIST, mean activation sparsity is essentially identical across depths (0.72, 0.72, 0.73), but PGD robust accuracy is 0.08, 0.57 and 0.40. Equal sparsity does hide different robustness, which supports the paper's premise that S(x) alone is insufficient.
  • Caveats we report in the paper:
    • The sign of the δ-free fixed-radius correlation on FashionMNIST goes from +0.22 (2 layers) to −0.01 (8 layers), so we claim no depth trend in the residual.
    • The 2-layer MNIST attack-point correlation is the lowest (0.80).
    • Robustness varies a lot across seeds (8-layer MNIST PGD accuracy ranges from 0.30 to 0.64 at λ = 0).
    • Per-layer instability shows no monotone early-versus-late pattern. The final FC layer has the lowest correlation in only 4 of 6 dataset-by-depth cells, so the 4-layer pattern in Appendix A is only partly reproduced.
  • Scope: two datasets, plain CNNs, two seeds. We have since added a single-seed ResNet-20 check on CIFAR-10/100 (Section 5.10): the coupling gap reproduces (attack-point 0.79–0.91 versus δ-free −0.33 to −0.38), but the δ-free residual is larger and S(x) correlates at +0.40 to +0.45 within each model, so the weak-sparsity finding is specific to the small CNNs. Attention architectures are not covered.

Q-D: Beyond ReLU. We agree this limits the scope of the numbers. M(x) = 1[a(x) > 0] needs a hard on/off boundary, and our limitations appendix says that D_M and A(x) do not transfer directly to GELU/Swish. Two points:

  • The sparsity literature we engage with (activation sparsity, energy/latency attacks) is about exact zeros, so ReLU-family networks are the natural setting for this question.
  • The coupling mechanism itself, a statistic evaluated at an attack-chosen point, is not ReLU-specific. What is ReLU-specific is the magnitude we measured, and we do not claim it generalizes. The revised title and abstract now say "ReLU networks", and the introduction says our magnitudes are measured only on small ReLU CNNs. We have not run a soft-mask variant for smooth activations. We list it as untested.

Presentation. We accept this criticism. The paper accumulated checks in response to its own caveats, and the structure showed it. In the revised PDF we have:

  1. Added explicit research questions (RQ1–RQ3) and a contribution list on the first page.
  2. Shortened the abstract and made it lead with the measurement-coupling point.
  3. Added a scope statement about what the paper does and does not claim.
  4. Condensed the Discussion and Limitations sections, which restated results at length. The extra depth material is in the appendix, so the main text stays within the page limit.

We recognize the secondary checks still make the paper dense, and we would welcome specific suggestions on what to cut or move.

Soundness. The summary lists soundness as the main concern. If the concern is the small architecture and the lack of depth variation, we hope the new depth experiment helps, and we do not claim generality beyond it ("a case study of one mechanism, not a general verdict"). If there is a specific flaw in the decoupling argument, such as the choice of δ-free measures or the statistics, we would be glad to address it. The decoupled measures do have their own design choices (K = 16 directions, subsampled conv layers), which we test in the appendix on robustness of the decoupled measures.

We thank the reviewer again and hope the clarified scope, the depth experiment and the revisions address the main concerns.

I acknowledge the rebuttal, and thanks authors for their explanation. I would still like to maintain my initial score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.