acceptodds
Under review as a conference paper at ICLR 2027

A Two-Scalar Law for Activation Functions: How Peak Slope and Peak Curvature Govern Adversarial Robustness

Abstract

Which intrinsic, measurable properties of an activation function govern adversarial robustness? We answer: exactly two scalars, in provably disjoint roles. For -layer fully connected networks with any twice-differentiable activation, we derive an exact path-summed decomposition of every diagonal element of the loss Hessian and prove that each element admits a closed-form upper bound affine in —the peak curvature—while every other activation-specific quantity in the bound reduces to the peak slope and to the prediction residual . This two-scalar law yields four a-priori, falsifiable predictions: (P1)once is large enough that fitting capacity is sufficient—so that the residual no longer inflates the bound—the normalized Hessian diagonal norm grows approximately monotonically with ; (P2)below this regime, the activation's fitting capacity becomes insufficient, which inflates the residual and drives a second rise of the same norm; (P3)in this same regime—where the residual no longer differs across variants—larger yields a provably larger bound, and correspondingly larger measured norms and faster robustness degradation, in one and the same ordering; (P4)activations of distinct functional forms share the same rise–peak–decline robustness profile along the common axis . Testing the law requires decoupling the two scalars; the Recursive Curvature-Tunable Activation Family (RCT-AF) provides exactly this, with scanning while stays fixed per variant independently of . All four predictions are confirmed across ResNet-18 and WideResNet-28-10, CIFAR-10, CIFAR-100, and TinyImageNet-200, and DAJAT, DKL, and TRADES under AutoAttack: robustness consistently attains its maximum in a common intermediate band of for every functional form and dataset, and large- degradation rates order by . These results reduce the design of robust activation functions to two measurable scalars.

Primary area:
learning theory
Keywords:
adversarial robustness, activation functions, loss landscape sharpness, Hessian analysis, adversarial training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hwz8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper bounds parameter-Hessian diagonal entries using an activation's peak slope and peak curvature, and tests the resulting empirical predictions about adversarial robustness using RCT-AF variants.

Critical strengths

The paper poses an interpretable question about activation properties and supports its investigation with controlled sweeps, multiple adversarial-training settings, and AutoAttack evaluation.

Critical weaknesses
  1. The assumptions and literature positioning do not sufficiently justify the breadth of the claims. Assumption 3.1 (lines 176–177) explicitly excludes ReLU-type activations, but the implications for the comparison with ReLU and the broader activation-design conclusions are not developed. In Section 2.3 (lines 129–135), “which measurable property carries the benefit” also presupposes a benefit from the smooth activations. I think one paper along from 2020 cannot justify this benefit in the first place. More discussion or at least the scope of the benefit should be discussed. In addition, for the whole section of related work, the related work is divided into topical subsections, but the discussion of activation-specific evidence remains too brief to support categorical statements such as “never isolate.” Please justify the smoothness restriction, qualify the benefit as setting-dependent, and provide a more systematic comparison with prior parametric-activation studies.
  2. Theorem 3.2 bounds curvature but does not establish the ordering of realized curvature or robustness. In lines 181–220, monotonicity applies to an upper bound with fixed weights and matched state factors. I think this alone does not imply monotonicity across separately trained networks. Although lines 270–276 and 440–447 acknowledge the empirical robustness bridge and possible slack, no quantitative tightness analysis is provided in the available main paper. In addition of experiment part, RCT-AF is the main controlled instrument, so the observed agreement could depend on its particular derivative proerties. I am wondering wheher the conclusion stay the same if we use a different activtion family as testbed?
Questions for authors
  1. Activation assumptions and claimed benefit (lines 129–135, 176–177): How should the conclusions be interpreted for ReLU, which is excluded by the standing assumption? The improvement in adversarial robustness cannot be justified as a general benefit of smoothness from a few cited settings. Please explain the restriction and state the architectures, training methods, and threat models for which the claimed benefit is supported.
  2. Bound tightness and empirical ordering (lines 181–220, 258–273, 440–447): How tight is Theorem 3.2 in the measured networks, and is its slack comparable across RCT-AF variants and unrelated activations? Please quantify and discuss the gap.
  3. Related work and generality (Sections 2.3–2.5): What distinguishes the proposed two-scalar explanation from prior studies that independently vary activation-shape properties, such as Dai et al. (2022)? Please organize the comparison by the property controlled, evaluation setting, and discuss whether the proposed ordering extends beyond the RCT-AF instrument.
  4. The appendix is inaccessible. It is not included at the end of the main text, and I could not open the supplementary material because of a “path too long” error. Since the appendix contains key proofs, I cannot fully assess the paper’s technical soundness without access to it. Please provide an accessible version.
2 comments
AuthorsOpenReview ↗

Response to Reviewer hwz8: ReLU as the alpha→infinity endpoint of the studied family (new Remark A.25), bound slack itemized (Remark A.23), structured prior-work comparison (Table 5), single-PDF appendix

We thank the reviewer for the careful reading. The revision addresses each question:

Q1 / W1 (ReLU and the claimed benefit). The resolution: the idealized ReLU is excluded by Assumption 3.1 because sigma'' = delta(x) at the kink (unbounded peak curvature), but ReLU is not outside our study — it is the alpha->infinity endpoint of the very family we sweep. The RCT-AF base function (beta = 0) is the parameterized softplus, the standard smooth surrogate of ReLU, with M2 = alpha/4 and pointwise convergence to max(x,0) as alpha grows (Appendix C.1). For every finite alpha, Theorem 3.2 applies verbatim, and its amplification term — affine in M2 — grows without bound along this endpoint. The law therefore predicts that past the band the amplification channel strengthens monotonically toward the ReLU extreme: inflated diagonal norms and robustness below the band optimum. This is what we measure: ReLU is dominated by the band configuration in all nine method-dataset pairs (Table 1), while remaining the strongest baseline in five of them (the law prescribes no total order off the band), and the measured norms rise monotonically for M2 > 10 along the family (Fig. 1a). The endpoint analysis is formalized in new Remark A.25: the ReLU comparison is thus not an out-of-scope anomaly but the extreme endpoint of our own sweep. Scope of the benefit: Sec. 2.3 now states the smoothness effect of Xie et al. (2021) as setting-dependent, and Sec. 4.6 / Appendix D.1 state precisely where the band-placement benefit is supported in this work: ResNet-18-class architectures, DAJAT+AWP/DKL+AWP/TRADES+AWP training, and the l-infinity eps = 8/255 threat model under AutoAttack. The categorical "never isolate" is replaced by the itemized comparison in Appendix F (Table 5).

Q2 / W2 (bound tightness). Three concrete additions. (i) Two caveats now follow Corollary 3.3 directly: the monotonicity concerns the bound at fixed network state — ordering across separately trained networks is an empirical finding, consistent with but not implied by the corollary — and the slack may be variant-dependent. (ii) Remark A.23 itemizes the slack analytically into three sources: (S1) triangle inequalities over path gains; (S2) the pointwise envelopes |sigma''| <= M2 and |sigma'| <= M1, which discard sign structure (sign cancellation can leave the realized |B| far below its bound); (S3) per-term bounding inside the coefficient, discarding inter-layer correlations. What the derivation guarantees — and the experiments confirm — is the channel structure (affine M2 growth, M1-powered coefficients, residual factor), each dominating a distinct regime (Appendix B). (iii) The underlying decomposition is exact (Theorem A.5) and verified numerically to floating-point precision for all depths and all beta (Remark A.9).

Q3 (Dai et al. 2022; beyond the RCT-AF instrument). Appendix F (Table 5) organizes the comparison along the requested axes: property controlled — prior studies vary shape descriptors and report PGD-based correlations, whereas we vary the measurable scalars M1 and M2 themselves, decoupled by construction; mechanism — reported correlations vs. a derived exact Hessian decomposition that predicts the U-shape, the band, and the M1-ordering; evaluation — PGD on CIFAR vs. two channels under AutoAttack across architectures, datasets, and methods. Notably, Dai et al.'s finding that moderate (not minimal) near-zero curvature helps is directionally consistent with our residual channel (M2 -> 0 is fitting-limited) — two independent lines of evidence agreeing in sign. Beyond the instrument (Sec. 3.5, Appendix D.1): P1, P2, and P4 require only that M2 be controllable along a shared axis — any such family suffices; the M1-ordering of P3 additionally requires the decoupling of Property 1. The law itself (Theorem 3.2) holds for any C2 activation, hence applies to any candidate family; the cross-family check in Table 1 (ReLU, GELU, Swish occupy precisely the positions the law assigns them) supports this reading, and a cross-family test of the M1-ordering is stated as future work.

Q4 (appendix accessibility). We apologize for the access failure. The revision is a single self-contained PDF: the complete Appendices A-F, including all proofs, follow the references. No external supplementary link is needed.

Thank you very much for your detailed response. Most of my concerns have been addressed. I will consider the other reviewers’ feedback and discuss the paper with the AC before deciding whether to raise my score.

Reviewer_oGBeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper how the maximum slope and curvature of activation functions relate to adversarial robustness.

Critical strengths

The paper demonstrates theoretical guarantees that look impressive.

Critical weaknesses

W1. Motivation of the paper needs to be clearer. In addition, motivation-related figures are needed to demonstrate what previous studies do not address and what this paper aims to address. W2. The paper lacks sufficient experimental analysis and results to validate the authors' claims and establish the approach effectiveness in real world settings. W3. Evaluation against diverse adversarial attacks and across different architectures is needed. W4. The presentation of the paper lacks clarity, which, in turn, making it difficult to understand the main motivation, contributions, what the proposed approach aims to achieve, and why it is needed.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oGBe: figure-supported motivation, expanded experiments (9 method-dataset pairs, Table 1), AutoAttack + 2 architectures + 3 datasets, restructured presentation

We thank the reviewer. The revision (a single integrated PDF: main text followed by complete Appendices A-F) addresses the four weaknesses:

W1 (motivation) + W4 (clarity). The motivation is now carried by a figure-supported passage at the top of Sec. 1: Fig. 2 (main text, Sec. 4.5) previews the gap this paper fills — under a fixed adversarial-training recipe, robust accuracy is unexplained by the raw activation parameter alpha (panel a), yet the three activation variants collapse onto nearly a single curve when replotted on the peak-curvature axis M2 (panel b). Isolating which measurable property produces this collapse — and proving it — is what prior holistic activation comparisons do not supply. Sec. 2 then organizes prior work by what each line leaves open: reliable evaluation settles how robustness is measured, not which architectural properties produce it; adversarial training methods are orthogonal to ours; smooth-activation studies identify no operative scale; sharpness work has no quantitative activation link; certificate analysis yields no activation-level law. Clarity fixes: a linear structure (Sec. 3 law -> Sec. 4 verification -> Sec. 4.6 activation comparison), a roadmap at the end of Sec. 1, a plain-language abstract, and all derivations consolidated in Appendix A.

W2 (experimental validation). The main-text evidence is substantially expanded: Table 1 (Sec. 4.6) covers nine method-dataset pairs (DAJAT+AWP / DKL+AWP / TRADES+AWP on CIFAR-10, CIFAR-100, and TinyImageNet-200), with the band configuration attaining the highest AutoAttack robust accuracy in every pair (+0.8-3.4 points over the strongest baseline) without sacrificing clean accuracy. All four predictions P1-P4 are verified in Secs. 4.2-4.5, and each observed phenomenon is mapped to a specific term of the same bound (Appendix B, Table 2). Appendix D.2 adds cross-setting generality (Figs. 6-8, Table 3), Appendix E quantifies computational overhead (beta = 0/1 cost like GELU-class activations), and the design recipe (beta in {0,1}, M2 ~ 4, no re-tuning) is stated in Sec. 4.6 and Appendix D.1.

W3 (evaluation reliability and attack diversity). All robustness numbers are measured with AutoAttack (Croce & Hein, 2020), the current de facto standard for adversarial robustness evaluation — adopted as the standardized evaluator of the RobustBench leaderboard (Croce et al., 2021), so our numbers are directly comparable with the literature. AutoAttack acquired this status precisely because it evaluates more accurately than traditional PGD- or C&W-style protocols: it is a parameter-free ensemble of four complementary attacks (APGD-CE, APGD-T, FAB-T, Square) whose components are empirically stronger than standard PGD, and it was designed to resolve the systematic overestimation of robustness that weaker evaluation produces — including gradient-masked defenses that PGD cannot break. On any fixed model, AutoAttack therefore reports lower robust accuracy than weaker protocols: it is the strictest standardized judge available, not a lenient one. Architectures: ResNet-18 and WideResNet-28-10 exhibit an identical M2 dependence (Appendix D.2, Fig. 6); training objectives: three (Fig. 7); datasets: three, including TinyImageNet-200 (Fig. 8).

Reviewer_pMrqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that adversarial robustness relates to two activation-function properties: (1) the maximum slope M1=max⁡z∣σ′(z)∣M_1 = \max_z |\sigma'(z)| and (2) the maximum curvature M2=max⁡z∣σ′′(z)∣M_2 = \max_z |\sigma''(z)|. It derives a loss-Hessian bound involving both and proposes RCT-AF (Recursive Curvature-Tunable Activation Family), a family of activations that vary the two properties more independently. Across several datasets, architectures, and adversarial training methods, the paper demonstrates that the highest robustness is achieved within an intermediate curvature range (M2∈[4,10]M_2 \in [4,10]), with slope influencing performance at the matched curvature.

Critical strengths
  • The paper provides a theoretical framework that derives a loss-Hessian bound that suggests how activation curvature (M2) and activation slope (M1) influence adversarial robustness.

  • The paper proposes RCT-AF as a control mechanism to more independently vary M1 and M2 in the experiments.

Critical weaknesses
  • The paper is very difficult to follow. It seems to be a rushed, last-minute submission. There are many technical terms that are yet obscure in the abstract and introduction (e.g., path-summed decomposition, a priori, falsifiable predictions, division of labor). The only experimental figure presented in the main paper is Figure 1, which is about standard training and clean accuracy. Figure 2 and Table 2 are included in the appendix; I believe both are important experimental results and should be presented in the main paper and discussed thoroughly. Generally, the paper is poorly structured, which made it very hard for me to follow the narrative.

  • The main theoretical result establishes an upper bound on each diagonal element of the loss Hessian, which does not directly translate to a bound on adversarial robustness. Also, the empirical evidence provided by the paper, in my view, is insufficient to establish a causal relationship between (M1, M2) and adversarial robustness.

  • Figure 2 looks interesting, which may potentially suggest a better activation design for improving robust accuracy of adversarially trained models. However, it is unclear how to set the hyperparameters for RCT-AF to achieve maximum performance. The experiments are also restricted to RestNet-18, CIFAR-10, and DKL. It is unclear whether similar patterns and optimal hyperparameter choices generalize across architectures, datasets, and adversarial training methods.

Questions for authors
  1. Table 2 says that all three adversarial training methods use adversarial weight perturbation (in its caption), but the three columns are labeled DAJAT, DKL, and Trades. This is confusing. Can the authors clarify which adversarial training methods are used?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pMrq: revised main text (Fig. 2 and Table 1 now in main), AT methods clarified (DAJAT+AWP/DKL+AWP/TRADES+AWP), evidence chain and cross-setting generality addressed

We thank the reviewer for the detailed comments. The revision (a single integrated PDF: main text followed by complete Appendices A-F) addresses each point:

1. Which adversarial training methods are used. The three columns are the three training objectives DAJAT, DKL, and TRADES, each run with adversarial weight perturbation (AWP) enabled by default — AWP is a composable weight-space regularizer, not a fourth method. To remove the ambiguity entirely, we renamed the columns DAJAT+AWP, DKL+AWP, TRADES+AWP everywhere they appear (abstract, Sec. 4.1, Table 1 and its caption, Fig. 2, Appendix D), and the Table 1 caption states this explicitly.

2. Presentation and structure. We rewrote the paper around a linear narrative: Sec. 3 states the law (Assumption 3.1 -> Theorem 3.2 -> Corollary 3.3 -> Remark 3.4 -> Proposition 3.5 -> the testbed in brief, Sec. 3.5); Sec. 4 verifies it prediction-by-prediction (Secs. 4.2-4.5); Sec. 4.6 reads the activation comparison. The phrases you flagged are gone: "path-summed decomposition" and "a priori" no longer appear in the abstract or introduction, the abstract states the four predictions in plain language ("stated before any experiment is run"), and "division of labor" is replaced by "two provably separate mechanisms ... in disjoint, individually provable positions" (Sec. 1). All derivations are consolidated in Appendix A with pointers.

3. Fig. 2 and the activation comparison are now in the main text. The robustness-vs-curvature figure (our Fig. 2) is in Sec. 4.5 and is discussed thoroughly there, including why panels (a)+(b) jointly establish P4 by elimination (alpha fails to explain the data; M2 is the only invariant across functional forms). The activation comparison (originally Table 2) is now Table 1 in Sec. 4.6: its reading is summarized in the main text, and the full discussion — dominance in all nine method-dataset pairs, margin structure, clean-accuracy ties, the ReLU analysis, and the design recipe — is in Appendix D.1.

4. The bound bounds Hessian diagonals, not robustness; causality. We agree, and the revision says so explicitly. Sec. 3.4 states the evidence chain: predictions are tested link by link — norm structure under standard training (P1-P3), then AutoAttack robust accuracy (P3-degradation, P4) — the last step inheriting the norm predictions through the empirically established flatness-robustness correspondence, "a bridge that is empirical and not part of Theorem 3.2." Sec. 4.1 replaces causal wording with: the sweep "establishes a controlled, falsifiable correspondence between (M1, M2) and robustness — the strongest attribution available without intervening on the network — rather than a formal causal claim." The strength comes from the controlled design: recipe fixed, only the activation scalars varied, every cross-form comparison at matched M2.

5. Hyperparameters and generality. Sec. 4.6 and Appendix D.1 give a concrete recipe: choose the cheapest form sustaining adequate fit (beta = 0 or 1; Appendix E shows both cost like GELU-class activations) and set M2 at the band start-point (M2 ~ 4). No per-dataset or per-method re-tuning was needed: the band endpoints were stable across all evaluated settings. Generality: Table 1 covers nine method-dataset pairs (the band configuration attains the highest robust accuracy in every pair, +0.8-3.4 points over the strongest baseline, without clean-accuracy cost); Appendix D.2 (Figs. 6-8, Table 3) varies one axis at a time — two architectures, three objectives, three datasets — with the same rise-peak-decline profile and band [4,10] in every setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.