- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper bounds parameter-Hessian diagonal entries using an activation's peak slope and peak curvature, and tests the resulting empirical predictions about adversarial robustness using RCT-AF variants.
Critical strengths
The paper poses an interpretable question about activation properties and supports its investigation with controlled sweeps, multiple adversarial-training settings, and AutoAttack evaluation.
Critical weaknesses
- The assumptions and literature positioning do not sufficiently justify the breadth of the claims. Assumption 3.1 (lines 176–177) explicitly excludes ReLU-type activations, but the implications for the comparison with ReLU and the broader activation-design conclusions are not developed. In Section 2.3 (lines 129–135), “which measurable property carries the benefit” also presupposes a benefit from the smooth activations. I think one paper along from 2020 cannot justify this benefit in the first place. More discussion or at least the scope of the benefit should be discussed. In addition, for the whole section of related work, the related work is divided into topical subsections, but the discussion of activation-specific evidence remains too brief to support categorical statements such as “never isolate.” Please justify the smoothness restriction, qualify the benefit as setting-dependent, and provide a more systematic comparison with prior parametric-activation studies.
- Theorem 3.2 bounds curvature but does not establish the ordering of realized curvature or robustness. In lines 181–220, monotonicity applies to an upper bound with fixed weights and matched state factors. I think this alone does not imply monotonicity across separately trained networks. Although lines 270–276 and 440–447 acknowledge the empirical robustness bridge and possible slack, no quantitative tightness analysis is provided in the available main paper. In addition of experiment part, RCT-AF is the main controlled instrument, so the observed agreement could depend on its particular derivative proerties. I am wondering wheher the conclusion stay the same if we use a different activtion family as testbed?
Questions for authors
- Activation assumptions and claimed benefit (lines 129–135, 176–177): How should the conclusions be interpreted for ReLU, which is excluded by the standing assumption? The improvement in adversarial robustness cannot be justified as a general benefit of smoothness from a few cited settings. Please explain the restriction and state the architectures, training methods, and threat models for which the claimed benefit is supported.
- Bound tightness and empirical ordering (lines 181–220, 258–273, 440–447): How tight is Theorem 3.2 in the measured networks, and is its slack comparable across RCT-AF variants and unrelated activations? Please quantify and discuss the gap.
- Related work and generality (Sections 2.3–2.5): What distinguishes the proposed two-scalar explanation from prior studies that independently vary activation-shape properties, such as Dai et al. (2022)? Please organize the comparison by the property controlled, evaluation setting, and discuss whether the proposed ordering extends beyond the RCT-AF instrument.
- The appendix is inaccessible. It is not included at the end of the main text, and I could not open the supplementary material because of a “path too long” error. Since the appendix contains key proofs, I cannot fully assess the paper’s technical soundness without access to it. Please provide an accessible version.
2 comments
Thank you very much for your detailed response. Most of my concerns have been addressed. I will consider the other reviewers’ feedback and discuss the paper with the AC before deciding whether to raise my score.
Response to Reviewer hwz8: ReLU as the alpha→infinity endpoint of the studied family (new Remark A.25), bound slack itemized (Remark A.23), structured prior-work comparison (Table 5), single-PDF appendix
We thank the reviewer for the careful reading. The revision addresses each question:
Q1 / W1 (ReLU and the claimed benefit). The resolution: the idealized ReLU is excluded by Assumption 3.1 because sigma'' = delta(x) at the kink (unbounded peak curvature), but ReLU is not outside our study — it is the alpha->infinity endpoint of the very family we sweep. The RCT-AF base function (beta = 0) is the parameterized softplus, the standard smooth surrogate of ReLU, with M2 = alpha/4 and pointwise convergence to max(x,0) as alpha grows (Appendix C.1). For every finite alpha, Theorem 3.2 applies verbatim, and its amplification term — affine in M2 — grows without bound along this endpoint. The law therefore predicts that past the band the amplification channel strengthens monotonically toward the ReLU extreme: inflated diagonal norms and robustness below the band optimum. This is what we measure: ReLU is dominated by the band configuration in all nine method-dataset pairs (Table 1), while remaining the strongest baseline in five of them (the law prescribes no total order off the band), and the measured norms rise monotonically for M2 > 10 along the family (Fig. 1a). The endpoint analysis is formalized in new Remark A.25: the ReLU comparison is thus not an out-of-scope anomaly but the extreme endpoint of our own sweep. Scope of the benefit: Sec. 2.3 now states the smoothness effect of Xie et al. (2021) as setting-dependent, and Sec. 4.6 / Appendix D.1 state precisely where the band-placement benefit is supported in this work: ResNet-18-class architectures, DAJAT+AWP/DKL+AWP/TRADES+AWP training, and the l-infinity eps = 8/255 threat model under AutoAttack. The categorical "never isolate" is replaced by the itemized comparison in Appendix F (Table 5).
Q2 / W2 (bound tightness). Three concrete additions. (i) Two caveats now follow Corollary 3.3 directly: the monotonicity concerns the bound at fixed network state — ordering across separately trained networks is an empirical finding, consistent with but not implied by the corollary — and the slack may be variant-dependent. (ii) Remark A.23 itemizes the slack analytically into three sources: (S1) triangle inequalities over path gains; (S2) the pointwise envelopes |sigma''| <= M2 and |sigma'| <= M1, which discard sign structure (sign cancellation can leave the realized |B| far below its bound); (S3) per-term bounding inside the coefficient, discarding inter-layer correlations. What the derivation guarantees — and the experiments confirm — is the channel structure (affine M2 growth, M1-powered coefficients, residual factor), each dominating a distinct regime (Appendix B). (iii) The underlying decomposition is exact (Theorem A.5) and verified numerically to floating-point precision for all depths and all beta (Remark A.9).
Q3 (Dai et al. 2022; beyond the RCT-AF instrument). Appendix F (Table 5) organizes the comparison along the requested axes: property controlled — prior studies vary shape descriptors and report PGD-based correlations, whereas we vary the measurable scalars M1 and M2 themselves, decoupled by construction; mechanism — reported correlations vs. a derived exact Hessian decomposition that predicts the U-shape, the band, and the M1-ordering; evaluation — PGD on CIFAR vs. two channels under AutoAttack across architectures, datasets, and methods. Notably, Dai et al.'s finding that moderate (not minimal) near-zero curvature helps is directionally consistent with our residual channel (M2 -> 0 is fitting-limited) — two independent lines of evidence agreeing in sign. Beyond the instrument (Sec. 3.5, Appendix D.1): P1, P2, and P4 require only that M2 be controllable along a shared axis — any such family suffices; the M1-ordering of P3 additionally requires the decoupling of Property 1. The law itself (Theorem 3.2) holds for any C2 activation, hence applies to any candidate family; the cross-family check in Table 1 (ReLU, GELU, Swish occupy precisely the positions the law assigns them) supports this reading, and a cross-family test of the M1-ordering is stated as future work.
Q4 (appendix accessibility). We apologize for the access failure. The revision is a single self-contained PDF: the complete Appendices A-F, including all proofs, follow the references. No external supplementary link is needed.