acceptodds
Under review as a conference paper at ICLR 2027

Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks

11 views

Abstract

Understanding why deep neural networks make a given prediction is of great importance for their safe deployment. In computer vision, saliency maps, which highlight the image region most influential for a prediction, remain a widely-used form of explanation. An empirical observation is the apparent sparsity of gradient saliency maps of adversarially-trained neural networks. In this paper, we propose a theoretical explanation of this phenomenon for two-layer ReLU networks. We build on the established equivalence of adversarial training to the minimization of the empirical risk with weight-decay penalization and an added adversarial total variation term – valid for certain loss functions. As the number of data points and neurons grows and the regularization parameters are sent to zero at appropriate rates, we prove that minimizers converge to a Bayes classifier with minimal gradient and Barron norm. Sparsity appears since for adversarial training with -attacks the gradient norm is anisotropic and favors axis-aligned / sparse gradients. We illustrate our theoretical findings experimentally by evaluating the gradient -norm and thresholded sparsity of naturally versus adversarially trained models.

Primary area:
learning theory
Keywords:
adversarial training, saliency maps, explainable AI, two-layer neural networks, Barron space, Gamma-convergence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GWbwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a variational explanation for sparse gradient saliency maps in adversarially trained networks. For two-layer ReLU networks with TV and weight-decay regularization, it claims that empirical minimizers converge, under prescribed joint scaling, to population-risk minimizers that optimally balance total variation and the Barron seminorm. For ℓ∞\ell_\infty perturbations, the resulting gradient ℓ1\ell_1 penalty is interpreted as promoting sparsity. Validation includes synthetic and image-classification experiments measuring accuracy, robustness, gradient norms, and thresholded sparsity.

Critical strengths

The variational formulation offers a conceptually useful connection between the perturbation norm, the induced gradient penalty, and network complexity. This provides a principled framework for investigating how adversarial regularization affects saliency geometry, supported by relevant qualitative illustrations in two-layer networks.

Critical weaknesses
  • Lemma A.5 applies a pointwise upper bound inside a Rademacher-weighted sum without handling negative signs (p. 18, lines 932–939). This invalidates the reduction used to derive the complexity estimate and leaves the required uniform empirical-TV concentration bound unestablished. Separately, Proposition 5.3 infers uniformly bounded weight decay from an upper bound on FnF_n (p. 30, lines 1584–1586), although the centered empirical-risk term can be negative and offset increasing regularization. No suitable lower bound is supplied to justify this inference. These gaps affect both the convergence and compactness arguments underlying Theorem 1 and are decisive for my recommendation.
  • Absolute-threshold sparsity can increase under uniform gradient shrinkage without any increase in relative concentration. Consequently, the reported quantitative metrics cannot establish concentration on fewer coordinates. Although the paper includes two-layer ReLU experiments and normalized qualitative visualizations, it lacks a scale-invariant quantitative concentration analysis and systematic tests of the prescribed joint scaling under conditions closely matching the theoretical assumptions (Section 6 and Appendix F). The connection between the proposed mechanism and the empirical phenomenon therefore remains insufficiently supported.
Questions for authors
  • Can the authors provide a valid replacement for the signed-sum argument in Lemma A.5, or an alternative uniform empirical-TV concentration bound sufficient for Theorem 1?
  • Additional evidence for the current claims: Do the sparsity trends persist under scale-invariant concentration metrics, and can the predicted behavior be demonstrated in a controlled two-layer ReLU setting following the joint scaling in Eq. (11)?
6 comments

We would like to thank the reviewer for their time and their careful review. We are happy that the reviewer finds our paper "clear and easy to follow" and acknowledges that it develops "a principled framework for investigating how adversarial regularization affects saliency geometry".

We are also very thankful that the reviewer correctly identified two mistakes in our proofs. We were able to fix both proofs and therefore restore the theoretical claims made in our paper under slightly stronger scaling assumptions.

Below we provide a point-to-point reply to the identified weaknesses and questions.

Weaknesses

  • Thanks for finding and pointing out the mistakes in Lemma A.5 and in Proposition 5.3. We were able to correct both and provide the proofs in the updated PDF.
    • We had already noted the mistake in Proposition 5.3 (which in fact also affects the proof of Proposition 5.1) and were able to fix it easily, using the standard generalization bound for the empirical risk from Lemma B.1 to get a lower bound.
    • The issue identified in Lemma A.5 is more substantial and we had to change our proof strategy to establish the result. The new strategy involves a bound for the covering number of Barron balls (new Lemma A.5), which is then leveraged to obtain a bound for the desired Rademacher complexity using Dudley's entropy integral (new Lemma A.6). The new bound slightly differs from the old one, as it includes an additional log⁡32(n)\log^\frac{3}{2}(n) term in the numerator. As a consequence also the main theorem's scaling assumptions slightly changed. We also corrected a step of the same type in the proof of Lemma A.3, whose statement is unaffected.
  • We agree that absolute thresholded sparsity on its own is not a sufficient metric to evaluate sparsity. To address this limitation, we have extended our evaluation to also include the scale-invariant Hoyer sparsity measure (Figure 11) for the models from Figure 3. The trend observed there is the same as for the thresholded sparsity: Increasing the adversarial training budget leads to higher sparsity. New ablations (Figure 12) indicate that this effect specifically relies on the ℓ∞\ell_\infty-adversary, as neither increasing the weight decay nor ℓ2\ell_2-adversarial training shows a similar behavior.

Questions

  • Yes, we were able to correct the proof of Lemma A.5 as well as establish a lower bound for the centered empirical risk.
  • To support our sparsity claims, we included the scale-invariant Hoyer sparsity measure and evaluated it for the models from Figure 3. We found that the trend that higher adversarial training budget leads to more sparsity is also reflected in this metric (see Figure 11 in the new version of the manuscript)

Thank you for the revisions. The revised proofs appear to address the two specific gaps identified in my original review, and the Hoyer results and ablations substantially address the uniform-gradient-shrinkage concern. However, two substantive concerns remain.

1. Existence of empirical minimizers (p. 4).

Weight decay does not guarantee attainment when the output bias is unregularized. For example, consider uniform inputs on (−1,1)(-1,1), independent balanced binary labels, and the clipped logistic loss

ℓ(s,y)=min⁡({log⁡(1+e(1−2y)s),4)}.\ell(s,y)=\min(\{\log(1+e^{(1-2y)s}),4)\}.

This satisfies the stated assumptions and has population-risk minimizer f⋆=0f^\star=0. On an all-positive sample, constant predictors fb≡bf_b\equiv b have zero TV and weight decay, while their empirical loss approaches zero as b→+∞b\to+\infty. Every finite predictor has strictly positive loss, so the infimum is not attained. Please qualify the existence claim and clarify whether the result relies on eventual attainment or quasi-minimizers.

2. The requested joint-scaling experiment remains missing.

My original question asked for both scale-invariant metrics and a controlled two-layer experiment following Eq. (11). The response addresses only the first part. The new Hoyer results concern deep ResNets with fixed weight decay, while the two-layer experiments use isolated configurations and threshold-based sparsity measurements. The connection between the observed concentration and the theorem’s specific selection mechanism therefore remains indirect.

The revisions improve the manuscript, but these concerns about its formulation and central explanatory claim remain unresolved. I would need clearer justification or qualification of these claims, together with a direct response to the outstanding experimental question, before supporting acceptance.

Thanks for your reply.

1. Existence of empirical minimizers: This is a valid point since indeed we did not discuss existence in our paper, similar to [1] who also analyzed weight decay without bias regularization. Note that indeed our results apply to quasi-minimizers as long as their energies are sufficiently small, see Remark 3.2.

The reviewer's example is very insightful but not generic. In fact, whenever at least two different labels exist, existence of solutions is restored. In fact, since P(yi=1 ∀i=1,…,n)=2−n\mathbb{P}(y_i=1\,\forall i=1,\dots,n)=2^{-n} by Borel-Cantelli the probability of this example occuring infinitely often is zero. In fact, existence even without bias regularization can easily be proved under a slight strengthening of Assumption 1 that essentially asks R^n(f)→∞\widehat{\mathcal R}_n(f)\to \infty as f→±∞f\to\pm \infty. This would be in conflict to the boundedness of the loss which is why we are currently working on removing the boundedness assumption altogether, just like in [1]. Then we can treat all the standard coercive loss functions like the logistic cross-entropy loss suggested by the reviewer, the square loss, zero-one loss, etc.

Here is a proof sketch for existence under the above coercivity assumption and for inputs in a bounded set Ω\Omega:

  • Take a minimizing sequence of parameters θk=(ak,wk,bk)\theta^k=(a^k,w^k,b^k). Thanks to the weight decay, (ak),(wk)(a^k),(w^k) are bounded sequences and w.l.o.g. (after passing to a subsequence) we assume they converge.
  • For bounded weights ww and inputs xx we have ReLU⁡(w⋅x+b)=ReLU⁡(w⋅x+max⁡(b,−C))\operatorname{ReLU}(w\cdot x+b) = \operatorname{ReLU}(w\cdot x + \max(b,-C)) for some contant C>0C>0 that only depends on those bounds. Hence, w.l.o.g. we can assume that the inner biases bikb_i^k for i=1,…,ni=1,\dots,n and k∈Nk\in\mathbb N are bounded from below.
  • Let I∞:={i∈{1,…,n}:lim sup⁡k→∞bik=∞}I_\infty:=\{i\in\{1,\dots,n\}:\limsup_{k\to\infty}b_i^k=\infty\} be the indices of neurons for which the inner bias diverges. Again, w.l.o.g. we can assume that these limsups are limits. Then for k∈Nk\in\mathbb N large enough and i∈I∞i\in I_\infty we have ReLU⁡(wik⋅x+bik)=wik⋅x+bik\operatorname{ReLU}(w_i^k\cdot x+b_i^k) = w_i^k\cdot x+b_i^k and we can express the sequence of minimizing networks fθkf_{\theta^k} as fθk(x)=b0k+∑i∈I∞aikbik+∑i∈I∞aikwik⋅x+∑i∉I∞aikReLU⁡(wik⋅x+bik).f_{\theta^k}(x) = b_0^k + \sum_{i\in I_\infty}a_i^k b_i^k + \sum_{i\in I_\infty} a_i^k w_i^k\cdot x + \sum_{i\notin I_\infty}a_i^k \operatorname{ReLU}(w_i^k\cdot x+b_i^k).
  • There are two cases: If the sequence Bk:=b0k+∑i∈I∞aikbikB_k:=b_0^k + \sum_{i\in I_\infty}a_i^k b_i^k for k∈Nk\in\mathbb N is bounded, then (after passing to a subsequence) w.l.o.g. lim⁡k→∞Bk=:B∈R\lim_{k\to\infty} B_k=:B\in\mathbb R exists and fθkf_{\theta^k} converges uniformly on Ω\Omega to a neural network with at most nn neurons (noting that affine functions on bounded Ω\Omega can be represented as a ReLU neuron plus a constant). Otherwise, it holds that (again up to a subsequence) Bk→∞B_k\to\infty or Bk→−∞B_k\to-\infty which implies that fθk(x0)→±∞f_{\theta^k}(x_0)\to \pm\infty for some fixed x0∈Ωx_0\in\Omega. Together, with the uniform Lipschitz continuity of fθkf_{\theta^k} (with Lipschitz constant just depending on the bounded parameters aika_i^k and wikw_i^k) we get that fθk→±∞f_{\theta^k}\to\pm\infty uniformly in Ω\Omega. This contradicts the property of being a minimizing sequence since by the coercivity assumption the loss diverges.

2. The requested joint-scaling experiment remains missing: That is correct and we are still trying to figure out how to best design such an experiment. First, we would like to emphasize, however, that the scaling assumptions in (11) are standard assumptions for generalization bounds in statistical learning theory, see also our answer to Reviewer bqBK. Furthermore, the scaling εn≳log⁡3nn\varepsilon_n\gtrsim\sqrt{\frac{\log^3 n}{n}} is already extremely good and most likely close to sharp, compared to the naive scaling εn≳(log⁡nn)1d\varepsilon_n\gtrsim\left(\frac{\log n}{n}\right)^\frac{1}{d} which ensures that every εn\varepsilon_n-ball around a training data point contains at least one other training data point with high probability. Here, dd is the feature dimension! Finally, finding an explicit solution to the limit problem FF is very hard which makes an experimental verification of the convergence to such explicit solution difficult. What we are planning to include in the next days is a evaluation of the Bayes risk and the total variation for solutions fθnf_{\theta_n} for different values of the sample size nn on the toy data set from Figure 2.

[1] Park, J., Pelakh, I., & Wojtowytsch, S. (2023). Minimum norm interpolation by perceptra: Explicit regularization and implicit bias. Advances in Neural Information Processing Systems, 36, 8714-8756.

Reviewer_bqBKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theoretical analysis of the relationship between adversarial training and the sparsity of gradient-based explanations (saliency maps). The main result concerns one-hidden-layer ReLU neural networks trained under (ℓ∞\ell_\infty) adversarial perturbations with balanced weight decay regularization, in an asymptotic setting where both the number of training samples and the network width increase.

The theoretical framework relies on measure-theoretic arguments, including Gamma-convergence, Barron seminorms, and total variation regularization, to establish a connection between adversarial training and an (L^1)-type penalty on the input gradient on the learned NN. An extension to multiclass classification is also presented.

The theoretical results are illustrated on synthetic datasets, while experiments on CIFAR-10 and ImageNet provide empirical evidence of similar behavior in deeper architectures.

Critical strengths

1/. Interesting theoretical contribution: The paper addresses an interesting problem and develops a substantial theoretical analysis using measure theory in the empirical risk setting. Although the proof is technically demanding, the authors provide detailed derivations and adapt several existing results to their setting for completeness in the appendices.

2/. The paper also clearly identifies its contributions relative to Bungert & Stinson (2024) and Morris & Murray (2026), and explicitly acknowledges the limitations of its theoretical framework.

Critical weaknesses

1/. Limited scope of the theoretical result While clearly stated in the limitation section, the theoretical analysis is restricted to one-hidden-layer ReLU networks under strong asymptotic assumptions: increasing dataset size and network width, with adversarial budget and weight decay decreasing at specific rates. This considerably limits the scope and practical significance of the theoretical contribution. A theoretical extension to Resnet networks, as suggested by the authors, would substantially strengthen the paper (unfortunately in this paper the extension to deeper architectures is only supported empirically, where the observed phenomenon is already well known).

2/ Proof verification: The theoretical analysis is long and relies on several previous results adapted to this setting. I was unable to verify all the technical arguments in depth within the review timeframe. A complete verification would require substantially more time, possibly making this work better suited to a journal format.

3/ Contribution of weight decay to the observed sparsity: In the setting of the paper (1 hidden NN with ReLU) the gradient ∇xf(x)\nabla_x f(x) can be computed analytically: Σi∣wix+bi>0ai.wi\Sigma_{i| w_ix+b_i>0} a_i.w_i . Thus the L2 penalty on the weights directly constraint the gradient (saliency). If we cannot easily link this constraint to a L1-norm constraint on the gradient (as the paper does), I guess that we can make a link with the evaluated "thresholded gradient sparsity" (which is the main objective in the experiments l441: "sparsity refers to the ratio of components in the gradient whose magnitude lies below the threshold") : If ∥wi∥2≤C\|w_i\|_2 \leq C,

then Sϵ(wi)=Card({j:∣wi,j∣>ϵ})≤C2ϵ2S_\epsilon (w_i) = Card(\lbrace j:|w_{i,j}|>\epsilon \rbrace) \leq \frac{C^2}{\epsilon^2} . Therefore, a bounded parameter norm can itself induce a low proportion of components above a fixed threshold as the dimension increases. It may worth evaluating if we can derive a stronger constraint with adversarial training hypothesis, and also it should be observed in your experiments (Figure 7).

Additional minor Remarks

  • L84 the main claim on theorem 1 should explicitly state that the result applies only to one-hidden-layer ReLU networks.
  • Figure 1: Theorem 1 does not directly predict the behavior of ResNet-50. The figure caption should instead be presented as empirical evidence supporting a possible generalization of the theorem.
  • Notations l107 : The indexing is inconsistent: bib_i starts at i=0i=0, whereas aia_i starts at i=1i=1.
  • Equation 6: The shorthand notation for the supremum and infimum is standard in functional analysis, but could be explicitly introduced for clarity.
  • Related work: Several papers may be added if relevant in the related work:
  • Adversarial training restricts gradients closer to the image manifold: Bridging adversarial robustness and gradient interpretability, Kim et al , ICLR 2019
  • Connecting Lipschitz-constrained neural networks with explainability: On the Explainable Properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective, Serrurier et al , Neurips 2023.
Questions for authors

1/. Generalization to deeper architectures: What are the main obstacles to obtaining similar results for ResNet architectures as expected in l320? Is it only a future work statement or could be analysed deeper in this paper?

2/. Ablation study on the role of adversarial training versus weight decay: Since the theorem 1 requires both the adversarial training and weight decay. Could you provide experiments that show that adv training without WD leads denser saliency maps. Inversely, clean training with stronger weight decay constraint.

3/ I would be interested in the authors' perspective on this possible alternative approach based on the analytical expression of the gradient.(Weakness 3)

3 comments

We would like to thank the reviewer for their time and their detailed review.

We are happy that the reviewer appreciated that our submission analyzes "an interesting problem and develops a substantial theoretical analysis using measure theory in the empirical risk setting".

Below we provide a point-to-point reply to the identified weaknesses and questions. We are available for further discussions during the rebuttal phase.

Weaknesses

  1. Regarding the scope of the theoretical results: It is correct that our theoretical results are for now restricted to one-hidden-layer ReLU networks. As also pointed out to Reviewer oVhH our analysis, nevertheless, goes far beyond the state-of-the-art in the literature which is limited to linear classifiers. A generalization to residual networks appears feasible but is beyond the scope of this work due to the even heavier theoretical machinery needed (likely based on [2]).

    We would like to point out though that our asymptotic assumptions on the dataset size, network width, adversarial budget and weight decay are not really strong but rather the necessary and standard parametric rates: For the standard excess risk of a one-hidden-layer network with mnm_n neurons trained on nn data points to go to zero it is well-known [1] that mnlog⁡nn→0\sqrt{m_n\frac{\log n}{n}}\to 0 needs to hold. Our scaling condition (11) is a generalization of that standard scaling to the case of adversarial training and weight decay that differs from the standard condition just by a log factor: Indeed, if mnlog⁡3/2nn→0\sqrt{m_n}\frac{\log^{3/2} n}{\sqrt n}\to 0 and additionally 1εnmn→0\frac{1}{\varepsilon_n\sqrt{m_n}}\to 0 we get that

    1εnlog⁡3/2nn=1εnmnmnlog⁡3nn→0\frac{1}{\varepsilon_n}\frac{\log^{3/2} n}{\sqrt n}=\frac{1}{\varepsilon_n\sqrt{m_n}}\sqrt{m_n\frac{\log^3 n}{n}}\to 0

    as required by (11). The fact that εn\varepsilon_n cannot go to zero too fast is required to not lose the effect of the adversarial regularization.

  2. We concede that the reviewing timeframe of top-tier ML conferences like ICLR makes it hard to impossible to verify all proofs. Still a lot of theoretical work with heavy proofs is regularly published in such venues. As authors, we take the final responsibility for the correctness of the proofs. (Note that two mistakes were detected by Reviewer GWbw which we could both fix).

  3. It is true that a small ℓ2\ell_2-norm of a vector -- and hence weight decay regularization -- can imply large thresholded sparsity. However, note that the (thresholded) sparsity in Figure 3 converges to to zero as ε→0\varepsilon\to 0 despite the weight decay remaining active. To further support our claims, we included an evaluation of the scale-invariant Hoyer sparsity measure (Figure 9) for the models from Figure 3.

Minor remarks:

Thanks for the additional remarks which we all incorporated. Note that we believe the indexing for the weights and biases on pages 2-3 is indeed correct. Since b0b_0 is the output bias, there are m+1m+1 biases and mm weights.

Questions

  1. As already indicated, an extension to ResNets is beyond the scope of this paper, but an interesting directioin for future work. While the flow induced space [2] provides approximation results and Rademacher bounds, results like the lower semicontinuity of the norm (Lemma A.2) or compactness (required for Proposition 5.3) appear to be challenging.
  2. We would like to note that the results visualized in Figure 3 can be seen as an ablation study for the role of adversarial training. For these experiments the weight decay parameter was fixed, while the adversarial budget was changing. We also added new results studying the influence of the weight decay parameter for clean training and of the adversarial budget when performing adversarial training with an ℓ2\ell_2-adversary in Figure 12 (Appendix F). The results indicate that the effect on sparsity of the gradients is mostly driven by adversarial training with an ℓ∞\ell_\infty-adversary.
  3. It is true that in the case of one layer ReLU networks bounds for the network parameters translate to bounds for the gradient norm and that this can be used to derive bounds for the thresholded sparsity. It is also correct that numerical experiments indicate stronger constraints on sparsity than the thresholded sparsity, see our new evaluation of the Hoyer sparsity measure. Moreover, our additional ablation study on the influence of the weight decay parameter (Figure 12 in the new version of the manuscript) indicates that in practice the observations from Figure 3 are not due to the weight decay.

[1] Barron, Andrew R. "Approximation and estimation bounds for artificial neural networks." Machine learning 14.1 (1994): 115-133.

[2] E, Weinan, Ma, Chao, and Wu, Lei. "The Barron space and the flow-induced function spaces for neural network models." Constructive Approximation 55.1 (2022): 369-406.

Thank you for the detailed response and additional experiments. I agree that the Hoyer sparsity measure is more appropriate than thresholded sparsity.

I also acknowledge that Reviewer GWbw identified mistakes in the proofs and that the authors have provided corrections to address them.

To end with, I still believe that the discrepancy between the theoretical setting and the experiments limits the scope of the conclusions. In particular, the influence of the adversarial budget ϵ\epsilon is demonstrated on ResNet architectures (Figure 3), whereas the theoretical analysis is restricted to one-hidden-layer networks width an increasing width and datset size.

Despite these remaining concerns, I maintain my initial assessment of weak acceptance, as I do not find the authors' response sufficient to justify a higher score.

Thank you very much for your response. It is true that our theoretical result does not directly predict the behavior of ResNet (or other complicated) architectures and we will make sure not to claim this anywhere.

The main contribution is really the theoretical analysis of the phenomenon for shallow ReLU networks which is technically very difficult and, to the best of our knowledge, goes far beyond existing theoretical results. In our empirical validation we decided to have:

  • experiments closely matching our theoretical setting (Figures 2, 4–9), which provide empirical evidence for the theoretical predictions
  • experiments on deep ResNets (Figures 1, 3, and 10–13), which investigate whether similar phenomena can also be observed in more complex and practically relevant architectures.

We believe these two types of experiments complement each other: while the former validate our theory, the latter demonstrate the broader relevance of the phenomenon.

Hence, we would respectfully like to argue that the experiments going beyond the scope of our theoretical analysis could be viewed as an additional contribution rather than as a limitation of the theoretical results themselves. We believe that observing similar behavior in architectures beyond our current analysis makes the theoretical contribution all the more valuable.

Thank you again for the active rebuttal phase and your helpful comments on the paper.

Reviewer_oVhHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically explains why neural networks which are trained adversarially produce sparser gradient-based saliency maps (which are used to explain the models). It studies and formulates adversarial training where the goal is to also minimize an additional metric i.e. nonlocal total variation (TV) regularization along with weight decay. The theoretical proofs are given specifically for 2-layer Neural Network. Further using gamma convergence, this paper establishes minimizers converging to classifiers which minimize a combination of their Barron semi-norm and an input-gradient regularizer. When using l-infinity perturbations, the regularizer translates to the l-1 norm for the input gradients, providing an explanation to saliency maps’ sparsity. It further provides empirical results to the proven theorems.

Critical strengths

The theoretical results are really impressive. It addresses and provides an explanation for an important previously observed connection between adversarial robustness and saliency-map sparsity. The theoretical results combine gamma convergence, barron space analysis, and generalization bounds to extend existing population-level, nonparametric results with finite-width neural networks.

Critical weaknesses

The theoretical guarantees are only for neural networks with a single hidden layer, for binary classification with bounded Lipschitz losses and very specific scaling of certain parameters. However most practical deployments use much deeper networks, solving multiclass classification problems. Even the loss function is not something which is usually used for even binary classification problems. This raises a serious concern about how the results are practically important. I would like the authors to address if it is possible to extend the results over a more generic set of neural networks as well as normal adversarial training (TV regularized training is not something done in practice extensively and they are not equivalent). Even the experiments do not agree with the theoretical results as the ResNet-18 and ResNet-50 models are much deeper and they are trained with cross entropy loss function. I also think the paper provides a weaker justification about how minimizing the average l-1 norm of gradients encourages sparsity. The primary sparsity measure counts gradient components below fixed thresholds. In appendix F, the paper acknowledges that most gradient components are near zero rather than exactly zero. I would additionally like to see a scale normalized comparisons to distinguish genuine sparsity. Furthermore the paper motivates sparsity with better model explainability however the experimental results do not attempt to validate these claims.

Questions for authors

See above

2 comments

We would like to thank the reviewer for their time and are happy that they find our "theoretical results [...] really impressive" and confirm that these provide an "explanation for an important [...] connection between adversarial training and saliency-map sparsity".

Below we provide a point-to-point reply to the identified weaknesses and questions. We are available for further discussions during the rebuttal phase.

Weaknesses and Questions

  • Regarding the scope of the theoretical guarantees: We already discuss this in detail in the paragraph Limitations and Extensions on pages 6-7. There is the following hierarchy: While the case of multi-class classification is a trivial extension of our results, a generalization to residual neural networks appears doable but beyond the scope of this work, and an extension to arbitrary deeper architectures seems very hard or impossible (in fact even for standard instead of adversarial training). We give more details below:
    • It is correct that our theoretical guarantees only apply to single hidden ReLU networks, however, this analysis already goes far beyond comparable results in the literature which are all restricted to linear classifiers. Even providing error bounds for standard empirical risk minimization over neural networks with more realistic architectures is a very difficult problem in the field of statistical learning theory. Some very recent work establishes quantitative approximation properties for Barron functions with deep networks, see [1,2]. Our work is part of a huge body of literature which provided theoretical guarantees of learning problems through a mean-field analysis of one-layer neural networks in Barron spaces.
    • Our Assumption 1 on the loss functions ℓ(u,y)\ell(u,y) is very general and, in fact, standard in statistical learning theory. As we discuss after the assumption it is satisfied for clipped versions of the usual choices like cross-entropy, hinge, ℓ1\ell_1, etc.
    • It is true that our analysis as of now only covers original adversarial training (AT) with the zero-one loss as well as TV regularized training for arbitrary loss functions. However, since TV regularization is a special case of the TRADES method, we think it is an interesting and already sufficiently challenging method to consider. Furthermore, as our numerical results show, it behaves very similarly to AT, both qualitatively and quantitatively, even for general loss functions.
  • Regarding the experiments: First, there seems to be a misunderstanding since indeed the cross-entropy loss fulfills our assumptions. Second, note the following hierarchy of experiments:
    • Figures 2 and 4-9: Two layer networks trained with cross-entropy loss, precisely being covered by our theory
    • Figures 1, 3, and 10-13: Deep ResNets trained with cross-entropy loss, indicating that our theoretical results appear to be true even for these architectures.
    • Lastly, the results of Zhang, Zhu, ICML 2019 indicate that our theoretical results even appear to be true for convolutional architectures.
  • Regarding sparsity and explainability: Our theoretical analysis is an asymptotic one, saying that if the training data is sufficiently large and the adversarial and weight decay regularizations are sufficiently small, classifiers converge to such with sparse gradients. No exact sparsity can be expected for finite data and positive regularizations which is in line with the experiments. We will carefully revise our claims in order to make sure we do not overclaim. We would also like to note that we have included an evaluation of the scale-invariant Hoyer sparsity measure in the new version of our manuscript (see Figure 11). The trend observable there is consistent with our previous observations of higher sparsity for the adversarially trained models.

[1] Liao, Yulei, and Ming, Pingbing. "Spectral Barron space for deep neural network approximation." SIAM Journal on Mathematics of Data Science 7.3 (2025): 1053-1076.

[2] Ma, Songqiu, and Yang, Yunfei. "Minimax rates for learning spectral Barron functions by deep ReLU neural networks." arXiv preprint arXiv:2609.39020 (2026).

AuthorsOpenReview ↗

Update: General loss functions are now included

This is just quick update regarding our assumptions on the loss function:

As an outcome of the discussion with Reviewer GWbw, we were able to completely remove the boundedness assumption of the loss function which, in particular, allows for most standard loss functions, in particular, the cross-entropy. See Assumption 1 and Example 3.1.

We would be happy to clarify any other concerns that the reviewer might have and are looking forward to your response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.