acceptodds
Under review as a conference paper at ICLR 2027

Which Geometry Predicts Generalization Depends on What You Vary — Activation or Optimizer

Abstract

Recent work introduced R\'enyi sharpness, a reparameterization-invariant measure of loss-Hessian geometry (Zhang et al., 2026), and reported that it correlates strongly and consistently with the generalization gap, the difference between training and test accuracy. That evidence, however, comes from varying the optimizer while the activation function is held fixed. We invert this setup. We fix the optimizer to stochastic gradient descent (SGD), with a transfer check under a sharpness-aware optimizer, and vary the activation function across seven common functions. Across multiple datasets and architectures, we compare weight-space, input-space, and activation-level measures by their signed Kendall rank correlation with the activation-induced generalization gap. In this setting, R\'enyi sharpness is weak and changes sign across architectures and datasets (mean Kendall of in our main comparison). In contrast, the input Jacobian, which measures how sensitive the output is to the input, and a mechanistically motivated activation-level quantity are positively associated with the gap in every setting of our main comparison (mean of each). This quantity, the negative-branch leak, is the mean squared activation slope on negative pre-activations, , where is the activation and the pre-activation. A controlled intervention that gates only the negative branch provides causal evidence that the leak contributes to input sensitivity. Guided by this mechanism, we control the leak in two ways. First, we reduce it directly by scaling down the activation's negative branch, with no added loss term. This improves clean accuracy for every activation in our main setting, by up to points, and by up to points on a wide residual network (WRN-28-10). Second, we add a leak penalty to the training loss, together with the established input-Jacobian penalty (Hoffman et al., 2019). The leak term mainly affects clean accuracy, while the input-Jacobian term drives corruption robustness. Which geometry predicts generalization depends on the axis along which models vary.

Primary area:
optimization
Keywords:
activation functions, neural network generalization, Rényi sharpness, loss-Hessian geometry, input Jacobian, negative-branch leak, loss landscape

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Hd4DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(−1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an interesting and relatively unexplored question: which geometric quantity can predict generalization differences when the optimizer is fixed and the activation function changes? The author found that Rényi sharpness is a reliable generalization prediction metric under optimizer changes, but becomes inconsistent under activation function changes. As an alternative, the author proposes an activation level called negative branch leakage.

Critical strengths

1.This paper discusses an interesting problem and provides an intuitive solution for negative branch leakage. 2.The activation functions covered by the experiment are sufficient, and there are seven common activation functions.

Critical weaknesses

1.The small sample size problem in correlation analysis weakens the statistical reliability of the core claims. 2.The evidence provided in the paper regarding the causal leakage to generalization is insufficient.

Questions for authors

1.Can the paper provide experiments on other commonly used optimizers, such as SAM, Adam, etc.? 2.The Kendall coefficient is not sufficient to support the conclusion. Can more evidence be provided to support the conclusion?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hd4D (Part 1/2)

We thank you for recognizing the problem as interesting and the activation coverage as sufficient. We address each point with new analyses and new training runs, all added to the revised PDF; table numbers refer to the revised version, and W/Q follow the numbering of your review.

Note on the correlation grid. While preparing this response, we checked every model of the correlation grid (Figure 1, Table 5; 73 models). Three had stopped before the 200-epoch schedule under an early-stopping rule: ReLU and SELU on CIFAR-10/ResNet-34 (epochs 11 and 84) and ReLU on CIFAR-100/ResNet-18 (epoch 17). In addition, for activations implemented in place, the leak had been read after the activation overwrote its input, and Mish was recorded as zero. We regret that these models were incomplete in the submitted version. The three models are retrained for the full 200 epochs, the leak is read on the pre-activation (checked against automatic differentiation), and Figure 1, Table 5 and every table built on the grid are recomputed (Appendix O). Table 5 now reports all 13 measures of Figure 1 on all 11 settings (a setting is one dataset and one architecture, trained once with each activation), including the measures that are as strong as or stronger than the leak. All statistics below are computed on this completed grid.

W1 and Q2. Sample size and evidence beyond Kendall's τ\tau. Each setting has seven models (five for ViT-B/16 and ResNet-50), so we added evidence at four levels.

  1. Exact tests per setting. For each of the 11 settings of Table 5, new Table 12 gives the exact permutation pp over all n!n! orderings, and new Table 11 Spearman's ρ\rho and a bootstrap 95% interval for τ\tau, for all 13 measures. The leak is significant and positive in 6 of the 11 settings, the input Jacobian in 4, and Rényi sharpness in none.
  2. All settings together. A grid-level permutation test over all 11 settings (73 models), shuffling gaps within each setting, tests the mean τ\tau (Section 6.2, Table 6):
Measuremean τ\tau (accuracy gap)95% interval over settingspermutation pp
Leak+0.49[0.18, 0.74]<0.001
Input Jacobian+0.36[0.03, 0.67]<0.001
Rényi sharpness+0.13[−0.08, 0.34]0.09 (n.s.)
  1. Robustness to the target. The same holds for the loss gap and for both gaps with training accuracy and loss recomputed at inference, without data augmentation (new Table 6): Rényi sharpness is not significant under any of the four targets (pp from 0.07 to 0.76), while the leak (mean τ\tau +0.36 to +0.55) and the input Jacobian (+0.22 to +0.37) are significant under all four (p≤0.02p \le 0.02).
  2. More seeds. We repeated the grid of our main setting, CIFAR-100/ResNet-18, with three seeds per activation (21 models, new Table 7). The seed changes the gap of an activation by 0.26 points (mean SD), against a spread of 5.04 points across the seven activations, so the ranking reflects the activation, not the seed. The leak is significant on every seed (τ\tau = +0.90, +1.00, +0.81; exact p≤0.005p \le 0.005), as is the input Jacobian (+0.81, +0.62, +0.81; p≤0.035p \le 0.035), while Rényi sharpness is significant on none (+0.24, +0.33, +0.52; p≥0.07p \ge 0.07).

The cross-axis predictor of Table 9 is also scored over 145 models (15 settings). Beyond correlation, the central evidence is interventional (W2) and does not depend on rank statistics.

AuthorsOpenReview ↗

Response to Reviewer Hd4D (Part 2/2)

W2. Causal evidence linking the leak to generalization. The paper tests the leak with three interventions that target it with the architecture, optimizer and schedule fixed, plus a new mechanism measurement. On ResNet-18, training accuracy exceeds 99.9% in every run of Tables 1 and 14, so there a higher test accuracy is a smaller generalization gap.

  1. Gating within an activation (Eq. 4). The coefficient aa scales only the negative branch, with architecture, optimizer and schedule fixed. With aa selected on validation and three seeds, reducing the leak improves clean accuracy over the original activation for the high-leak activations on CIFAR-10 (ELU +1.03, SELU +1.23; Table 14) and on CIFAR-100 (ELU +1.53, SELU +2.45; Table 1). Closing the branch (a=0a=0) also raises accuracy on WRN-28-10 (SELU +10.85, ELU +8.06, three seeds each; Table 13), where ELU and SELU at a=1a=1 do not fit the training set (82 to 91%, Table 13), so there the gain includes better fitting.
  2. Leak versus slope jump (Table 2). A Softplus β×a\beta \times a grid varies the leak at a fixed slope jump. Accuracy follows the leak (r=−0.96r=-0.96 over nine cells, three seeds each); a joint fit attributes the effect to the leak (t=−9.4t=-9.4), not to the jump.
  3. Training-time control (Eq. 5). Penalizing the leak during training, with the activation unchanged, improves clean accuracy under standard SGD in three of four architecture-dataset pairs (Table 21).
  4. New: where the effect acts. Layer-by-layer measurement on 38 trained networks (two seeds) shows that closing the branch lowers the input Jacobian by 33 to 52% for ELU and SELU through the learned layers of the network (Table 16, Appendix G.1).

Together these support a causal role of the leak; it is not the only factor (Appendix L).

Q1. Other optimizers (SAM, Adam). The paper already evaluates five sharpness-aware optimizers in addition to SGD: SAM, ASAM, RSAM, FSAM and SSAM (Table 21; three seeds, two in four cells; four architecture-dataset pairs), and repeats the gating intervention under RSAM (Table 18), where it reproduces the SGD pattern. For the correlation study, we trained the full activation axis (seven activations, CIFAR-100/ResNet-18) under SAM (ρ=0.05\rho=0.05) and under AdamW (Adam with decoupled weight decay), and recomputed every measure (Kendall τ\tau, accuracy gap; exact pp in parentheses):

OptimizerRényiLeakInput JacobianTraceParam. norm
SGD (mean of 3 seeds)+0.37+0.90+0.75+0.71+0.75
SAM+0.52 (0.07)+1.00 (<0.001)+0.81 (0.005)+0.81+0.81
AdamW−0.33 (0.88)+0.05 (0.50)+0.62 (0.035)+0.71+0.14

Under SAM the SGD pattern holds and is slightly stronger. Under AdamW the activation changes the gap much less (the seven gaps span 2.1 points, against about 5 under SGD), so the ranking is less determined; the input Jacobian and trace remain significant, the leak does not (+0.05), and Rényi sharpness again does not track the gap. The finding, that Rényi sharpness does not track the activation-induced gap while an input-space measure does, therefore holds under all three optimizers. Conversely, when the optimizer is varied (the six optimizers of Table 21, 72 models), the same Rényi estimator (α=1.5\alpha=1.5) tracks the gap strongly (mean τ\tau +0.67, p<0.001p<0.001): the contrast is between axes of variation. Both are new tables in Appendix F (Tables 7 and 8).

Reviewer_ucgyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the Renyi sharpness metric under activation layer variations and shows that it is a less consistent predictor of generalization gap compared to looking at the metric under optimization algorithm variations. The paper then proposes that scaling down the negative-branch leak of the activation layer improves clean accuracy. This is shown by introducing a leak coefficient and studying the effect of that on the test accuracy. Furthermore, they show that adding a leak penalty as well as an input-output sensitivity penalty improves overall clean accuracy as well as the corruption robustness.

Critical strengths
  1. Overall, the paper is well-written and self-contained, effectively including the most relevant literature. The paper is very easy to follow, and the results are clearly structured.
  2. The results are validated across diverse empirical settings, which provides practical significance to the paper's claims. Overall, the paper presents insightful experimental findings.
Critical weaknesses
  1. The paper shows that for some architectures, lower leak coefficients are better (a near zero), but for some architectures, a small leak (a non-zero leak coefficient) is the optimum. This is provided without any explanations or any intuitions or discussions into why it is the case? This is especially important because the two architectures are inherently very similar (ResNet-18 vs ResNet-20) so one would expect similar behaviors.
  2. Figure 1 suggests that there is no universal generalization gap predictor across all the metrics. How can we elevate the findings of this experiment to propose a combination of metrics in order to have a predictor that works in various settings including optimization variant and activation function variant. Could a combination of sharpness and Jacobian solve the problem?
  3. The paper would benefit from a deeper interpretation of the results. Providing new insights and intuitions derived from the empirical findings would significantly strengthen the paper.
Questions for authors

Why was the leak function used as the regularization in Equation (5) as opposed to the leak coefficient from Equation (4). The rationale behind this is not clearly stated.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ucgy ( Part 1/2)

We thank you for the positive assessment of the paper's clarity, scope and empirical breadth. All new results are in the revised PDF; table numbers refer to the revised version, and W/Q follow the numbering of your review.

Note on the correlation grid. While preparing this response, we checked every model of the correlation grid (Figure 1, Table 5; 73 models). Three had stopped before the 200-epoch schedule under an early-stopping rule: ReLU and SELU on CIFAR-10/ResNet-34 (epochs 11 and 84) and ReLU on CIFAR-100/ResNet-18 (epoch 17). In addition, for activations implemented in place, the leak had been read after the activation overwrote its input, and Mish was recorded as zero. We regret that these models were incomplete in the submitted version. The three models are retrained for the full 200 epochs, the leak is read on the pre-activation (checked against automatic differentiation), and Figure 1, Table 5 and every table built on the grid are recomputed (Appendix O). Table 5 now reports all 13 measures of Figure 1 on all 11 settings (a setting is one dataset and one architecture, trained once with each activation), including the measures that are as strong as or stronger than the leak.

W1. Why a small leak on ResNet-18 but not on ResNet-20. We re-examined this with validation-selected coefficients and three seeds. The validation search selects the same a=0.013a=0.013 on ResNet-18 for CIFAR-10 and CIFAR-100. At this value a small leak and the closed branch (a=0a=0) are within 0.25 points of each other for every activation on CIFAR-10 (revised Table 14), and both improve on the original high-leak ELU (+1.03) and SELU (+1.23). The per-layer measurement of Appendix G.1 (Table 16, on CIFAR-100) shows why: at a=0.013a=0.013 the trained network's input Jacobian and layer gains are nearly the same as at a=0a=0, so a leak this small acts as a closed branch. ResNet-20 behaves the same way: on CIFAR-100 its validation curve over aa is flat (about 69.0 across the searched range, Appendix G), and under RSAM its selected a=0.026a=0.026 is at most 0.04 points above the better of a=0a=0 and a=1a=1 (Table 18). So on both architectures the interior gives no reliable gain over the better endpoint; the interior optimum in the submitted version came from a test-set grid of aa values (now Table 17, reported for completeness and not used for selection). What differs across architectures is the size of the gain at a=0a=0. On CIFAR-100 it grows with the leak of the activation and, in our three architectures, with model size. Although both are residual networks, ResNet-18 (11.2M parameters) is about 40 times larger than ResNet-20 (0.28M), and WRN-28-10 has 36.5M. The gain at a=0a=0 is +0.86, +2.27 and +10.85 for SELU on ResNet-20, ResNet-18 and WRN-28-10, and −0.02, +1.40 and +8.06 for ELU (Tables 1 and 13). One reading, consistent with Table 16, is that a larger network has more room to adapt its learned layers around the activation; on WRN-28-10, ELU and SELU at a=1a=1 also do not fit the training set (82 to 91%, Table 13), so there the gain includes better fitting. The three architectures also differ in width, weight decay and batch size, so we state this as an observation, not a rule. Section 4 now has the paragraph "A small leak and the closed branch perform alike".

AuthorsOpenReview ↗

Response to Reviewer ucgy ( Part 2/2)

W2. A combined predictor across the optimizer and activation axes. We built one predictor over both axes and scored it on held-out settings (new Table 9). Activation axis: the 11 settings of Table 5 (SGD; ReLU, LeakyReLU, GELU, SiLU, Mish, ELU, SELU; 73 models). Optimizer axis: ResNet-20 and ResNet-56 on CIFAR-10 and CIFAR-100 with GELU, trained with SGD, SAM, ASAM, RSAM, FSAM and SSAM, three seeds each (72 models). A linear predictor is fit on 14 settings, with features standardized within each setting, and scored on the held-out setting by Kendall τ\tau. Mean held-out τ\tau, accuracy gap:

Measures combinedActivation axis (11)Optimizer axis (4)All 15 settings
Rényi sharpness (α=1.5\alpha=1.5)−0.04+0.58+0.13
Rényi sharpness (α=0.5\alpha=0.5)+0.15+0.16+0.15
Trace+0.50+0.22+0.43
Parameter norm−0.62+0.71−0.26
Input Jacobian+0.37+0.63+0.44
Leak+0.49+0.40+0.46
Rényi + Jacobian+0.33+0.64+0.41
Jacobian + Leak+0.32+0.63+0.40

To your question: combining sharpness with the Jacobian does give a predictor that works on both axes, but the Jacobian carries it. Adding Rényi sharpness does not improve on the Jacobian alone (+0.41 against +0.44 over all 15 settings), and the fitted weights put most of the weight on the Jacobian (0.60 against 0.07 for Rényi sharpness). No weight-space measure is strong on both axes: Rényi sharpness predicts only on the optimizer axis, trace falls from +0.50 to +0.22, and parameter norm reverses sign between the axes (one fitted weight cannot match both signs, which is why it is −0.62 here although its per-setting τ\tau on the activation axis is positive, Table 5). The leak and the input Jacobian predict on both. The loss gap gives the same ordering (Table 9), and Section 6.2 now states this result.

W3. Deeper interpretation. The new analyses give an interpretation of the main finding, now in a short "Interpretation" paragraph in the Discussion (Section 7).

  1. Why the axis matters. Changing the optimizer changes where training ends in weight space, which is what Hessian-spectrum measures such as Rényi sharpness summarize. Changing the activation changes how input variation propagates through the negative branch. Rényi sharpness has no significant association with the activation-induced gap under any of four targets (accuracy and loss gaps, also with training accuracy and loss recomputed at inference, without data augmentation; permutation pp from 0.07 to 0.76), while the leak and the input Jacobian are significant under all four (Table 6).
  2. How the leak acts. Measuring gain layer by layer on trained networks, we find that closing the negative branch raises the gain of the activation layers themselves, but the network then learns smaller gains in its convolution and normalization layers, and the net input Jacobian falls by 33 to 52% for ELU and SELU (Section 3, Table 16). The leak shapes what the network learns; it does not act only through the activation's own slope.
  3. What this means for design. The leak's analytic value under a standard-normal pre-activation is computable before training and orders the activations as the measured leak does (Appendix C.1), so it can be used to choose or modify an activation, while penalizing the input Jacobian during training improves robustness. These are the two roles separated in Sections 4 and 6.5.

Q1. Why Eq. (5) penalizes the leak and not the coefficient aa. The coefficient aa in Eq. (4) is an architectural constant: it is fixed before training and does not depend on the weights, so it cannot be penalized by gradient descent. Making aa learnable would turn the gate into a new parametric activation, which we do not propose (Section 4). The leak ℓ\ell in Eq. (3) depends on both the activation and the pre-activations zlz_l that the network actually produces (for SELU, the measured leak ranges from 1.08 to 2.74 across the 11 settings, Appendix C.1). Penalizing ℓ\ell therefore lets training reduce the leak through the weights, with the activation unchanged. This has three practical consequences: (i) the penalty applies to any activation and architecture without modifying the activation; (ii) it is compatible with any optimizer (six tested, Table 21); (iii) it targets the same quantity that the analysis measures, so the two controls, architectural (aa) and training-time (ℓ\ell), act on one mechanism through different routes. We added this rationale in Section 5, after Eq. (5).

Reviewer_MfFrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals which geometric measure predicts generalization gap depends on the source of model variation. When fixing optimizers and varying activation functions, the previously proposed Rényi sharpness loses consistent predictive power. The authors define negative‑branch leak as an activation‑level metric and then establish causal evidence linking higher leak to elevated input‑output sensitivity and larger generalization gaps. Two practical ways to regulate leak are proposed to enhance the generalization: (1) an architecture‑only knob scaling activation’s negative branch for improved clean accuracy without extra loss terms, and (2) a leak‑penalized training objective. Numerical results show that the leak penalty mainly boosts clean‑set accuracy, while input‑Jacobian regularization drives corruption robustness, with the two penalties acting complementarily. The authors' findings caution against seeking a universal weight‑space geometric predictor for generalization.

Critical strengths

The paper shows generalization‑predicting geometric metrics depend on which model component is changed. Rényi sharpness performs well for optimizer variations but fails for activation variations. Controlled gating experiments establish causal evidence for the negative‑branch leak mechanism, decoupling it from confounding slope‑jump effects. Past research on generalization‑oriented geometry has largely focused on the Hessian in weight space. This paper encourages greater attention to how geometric properties inside activation layers affect generalization, the idea is interesting.

Critical weaknesses
  1. The Rényi comparison is not on equal terms with Zhang et al. Section 2 says the accuracy gap matches their protocol, but Zhang et al. use the gap between test and training loss. Training accuracy is 100% here, so the target is effectively test error. Test error need not rank models the same way as the loss gap when calibration changes across activations. Zhang et al.'s invariance argument, and most of their strong correlations, are for layer-wise Rényi sharpness. This paper only computes the full-network version, which they already report as weaker.

  2. The evidence for the leak as a predictor is thinner than the abstract suggests. Figure 1 shows 5 of the 11 settings. In the full Table 5, the leak and the Jacobian are each negative in three settings, all of them outside Figure 1. Both average around +0.37, which is no better than untuned trace or parameter norm. More basically, App. C.1 says the measured leak matches its N(0,1) value within 3%. So in every setting the leak is just a fixed ordering of the seven activations, and its ττ tests whether the gap follows that ordering, not a property of the trained model.

  3. The mechanism and the practical gains are both overstated. Under the standardization actually used, gating increases the per-layer E[φ′2]E[φ′²] for every activation. For SELU it goes from roughly 1.07 to 1.47 by my calculation. The argument in Section 3 would therefore predict the Jacobian to go up, not down. Gating lowers the Jacobian but hurts CIFAR-100-C accuracy (Table 11), while the Jacobian penalty lowers it and helps. That does not fit input sensitivity as the mediator. On the practical side, the large gains are for ELU/SELU, which at a=0a = 0 are ReLU, and no gated variant beats ReLU on WRN-28-10.

Questions for authors
  1. On your existing checkpoints, does Rényi sharpness still look weak if you correlate against the loss gap or use the layer-wise version? With the same estimator, do you recover a strong correlation on the six-optimizer models of Table 13?

  2. How was the ReLU reference trained (standardization, weight decay)? Why does ELU at a=0a = 0 diverge on WRN-28-10 when SELU at a=0a = 0, which is the same function, does not?

  3. Which values of a were selected on validation and which on test? Could you re-report Table 7 with validation-selected a and multiple seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MfFr (Part 1/3)

We thank you for a careful and technical review. All new analyses below are in the revised PDF; table and figure numbers refer to the revised version, and W/Q follow the numbering of your review.

Note on the correlation grid. While preparing this response, we checked every model of the correlation grid (Figure 1, Table 5; 73 models). Three had stopped before the 200-epoch schedule under an early-stopping rule: ReLU and SELU on CIFAR-10/ResNet-34 (epochs 11 and 84) and ReLU on CIFAR-100/ResNet-18 (epoch 17). In addition, for activations implemented in place, the leak had been read after the activation overwrote its input, and Mish was recorded as zero. We regret that these models were incomplete in the submitted version. The three models are retrained for the full 200 epochs, the leak is read on the pre-activation (checked against automatic differentiation), and Figure 1, Table 5 and every table built on the grid are recomputed (Appendix O). Table 5 now reports all 13 measures of Figure 1 on all 11 settings (a setting is one dataset and one architecture, trained once with each activation), including the measures that are as strong as or stronger than the leak. Below, τ\tau is the mean over the 11 settings unless stated.

W1 / Q1. Loss gap, layer-wise Rényi sharpness, and a positive control. Loss gap. The accuracy gap remains the primary target, as in the submitted version. Following Zhang et al., we add the loss gap, and both gaps with training accuracy and loss recomputed at inference without augmentation ("re-evaluated" gaps), which addresses the saturation of training accuracy. Mean τ\tau (one-sided permutation pp of the mean):

MeasureAcc. gapLoss gapRe-eval. acc. gapRe-eval. loss gap
Rényi sharpness+0.13 (0.09)−0.07 (0.76)+0.15 (0.07)−0.07 (0.76)
Input Jacobian+0.36 (<0.001)+0.22 (0.02)+0.37 (<0.001)+0.29 (0.001)
Leak+0.49 (<0.001)+0.36 (<0.001)+0.55 (<0.001)+0.43 (<0.001)

Against the loss gap, Rényi sharpness is the only one of the 13 measures with a negative mean. Table 6 gives all 13 measures under the four targets, and new Figure 5 repeats the 11-setting grid for the loss gap. If you prefer, we can show the loss-gap grid (Figure 5) in the main text, next to Figure 1 or in its place. In the submitted version, "matching the correlation protocol of Zhang et al." referred to how the correlation is computed: the same five settings and Kendall τ\tau computed per setting over the trained models. The target differed, and we thank you for pointing out this ambiguity. Section 2 now names the loss gap of Zhang et al. and no longer states that the accuracy gap matches their protocol.

Layer-wise Rényi sharpness. As you note, the invariance argument and most of the strong correlations of Zhang et al. are for the layer-wise measure, so we added it. We implemented it (Rényi sharpness of each layer's Hessian block, Definition 2.1 of Zhang et al.) with the same estimator and data as the global one, verified it against an exact Hessian and PyHessian (Yao et al., 2020), and computed it for every convolution and linear layer of the 63 models of the nine settings with all seven activations. Layer-averaged, its mean τ\tau is +0.05 for the accuracy gap (p=0.34p=0.34) and +0.17 for the loss gap (p=0.05p=0.05); its sign varies across settings (−0.71 to +1.00), and only 7 to 8% of layers have a significant positive τ\tau. On the same settings the leak averages +0.46 and +0.30 (new Table 10).

Six-optimizer models, same estimator. On the 72 models of Table 21 (Table 13 in the submitted version: six optimizers, four architecture-dataset pairs, three seeds), Rényi sharpness (α=1.5\alpha=1.5) recovers a strong correlation: mean τ\tau +0.67 over the 12 groups of six optimizers (p<0.001p<0.001; +0.51 to +0.78 per pair), and +0.66 for the loss gap (new Table 8). The estimator detects the association when the optimizer varies; the difference lies in the axis of variation.

AuthorsOpenReview ↗

Response to Reviewer MfFr (Part 2/3)

W2. Strength of the leak as a predictor. Figure 1 settings. Figure 1 uses the five settings of Figure 3 of Zhang et al., so the two figures can be compared directly; the caption now says so. Table 5 reports the same 13 measures on all 11 settings (accuracy gap), and Figure 5 for the loss gap.

Negative settings. In the three settings where the leak and the Jacobian are negative (ResNet-20 on CIFAR-10 and CIFAR-100, ResNet-56 on CIFAR-100), trace, Fisher–Rao, PAC-Bayes, SAM-ℓ2\ell_2, ASAM-ℓ2\ell_2 and ASAM-ℓ∞\ell_\infty are negative as well (Table 5), so these settings are difficult for the weight-space measures too. New Table 12 gives the exact per-setting pp for all 13 measures: the leak is significant and positive in 6 of the 11 settings, the input Jacobian in 4, and Rényi sharpness in none.

Trace and parameter norm. On the completed grid the leak averages +0.49 (accuracy gap; about +0.37 in the submitted version, see the note in Part 1). Trace (+0.51) and parameter norm (+0.62) are as strong as or stronger than the leak on the activation axis, and Section 6.2 now states this. They do not carry over to the optimizer axis: on the 72 six-optimizer models (Table 8), parameter norm reverses sign and trace falls, while the leak and the input Jacobian keep their sign and remain at least as strong.

Mean τ\tau (acc. gap)Activation axis (11 settings)Optimizer axis (Table 8)
Rényi sharpness+0.13+0.67
Trace+0.51+0.29
Parameter norm+0.62−0.82
Input Jacobian+0.36+0.70
Leak+0.49+0.48

On the activation axis, parameter norm follows the leak: within each setting it orders the activations almost as the leak does (mean τ\tau +0.66). In the five-measure predictor of Table 9, its weight is 0.01, against 0.55 for the Jacobian and 0.32 for the leak.

This is the claim of the paper (Section 6.2): which geometry predicts generalization depends on the axis of variation, and this holds for weight-space measures beyond Rényi sharpness. We do not claim that the leak outperforms every weight-space measure on the activation axis; the comparison in the abstract is with Rényi sharpness.

Fixed ordering and App. C.1. The 3% statement in App. C.1 refers to the gated, standardized networks of Table 14, and the revised C.1 says so. In the Table 5 grid, the measured leak orders the seven activations the same way in all 11 settings (ReLU = LeakyReLU < GELU < SiLU < Mish < ELU < SELU), while its values vary with the trained network (SELU from 1.08 to 2.74). So, as you note, the leak τ\tau in Table 5 tests whether the gap follows this order. That the order is known before training is what makes the leak usable as a design rule. On the optimizer axis (Table 8), the activation is fixed (GELU), so the leak varies only through the trained pre-activations, and it still tracks the gap (τ=+0.48\tau=+0.48, p<0.001p<0.001). The test that the leak acts within a trained model is the intervention, where the architecture is fixed and the leak is varied through β\beta and aa: in the Softplus β×a\beta \times a grid (Table 2, ResNet-18/CIFAR-10), accuracy follows the measured leak (r=−0.96r=-0.96 over nine cells), and a joint fit attributes the effect to the leak (t=−9.4t=-9.4), not to the slope jump.

W3. Mechanism under standardization. Your calculation is correct for a standard-normal pre-activation: with standardization, gating raises the per-layer E[φ~′2]\mathbb{E}[\tilde\varphi'^2] (SELU 1.07 to 1.47). To show where the Jacobian decrease comes from, we measured input sensitivity layer by layer on the trained networks (CIFAR-100/ResNet-18, 1,000 test images, Jacobian-vector products along random input directions, two seeds, 38 models; two independent estimates of ∥J∥F\|J\|_F agree within 1.3%, and the per-layer terms sum exactly to log⁡∥Jv∥\log\|Jv\|).

Activation∥J∥F\|J\|_F, a=1a=1∥J∥F\|J\|_F, a=0a=0activation layers, ∑log⁡\sum\log gainall other layers, ∑log⁡\sum\log gain
ELU173115−0.39 → +2.49+2.90 → −0.03
SELU236114+0.46 → +2.48+2.26 → −0.02
GELU8390−1.47 → −0.10+3.56 → +2.40
SiLU10196−1.52 → −0.11+3.83 → +2.57
Mish97100−1.43 → +0.41+3.58 → +1.97

(seed 1; seed 0 agrees within a few percent)

As you compute, closing the branch raises the gain of the activation layers. The trained network lowers the gain of its learned layers (convolutions, normalization, residual paths) by more, so the input Jacobian falls by 33 to 52% for ELU and SELU. For GELU, SiLU and Mish, whose leak is small, it changes by less than 10% (GELU rises by 9%). The leak acts through what the network learns, not only through the activation's own slope. Real pre-activations are also not standard normal (fraction negative 0.55 to 0.75). The revised Section 3 states the mechanism at this level; Appendix G.1 adds the decomposition (new Table 16).

AuthorsOpenReview ↗

Response to Reviewer MfFr (Part 3/3)

W3. CIFAR-100-C. The CIFAR-100-C half of Table 19 (Table 11 in the submitted version) now has three seeds per cell, each trained for 200 epochs (seed 0 only in the submitted version, where the average change was −1.81; mean corruption accuracy, %, ResNet-18):

ELUSELUGELUSiLUMishLeakyReLU
a=1a=121.27 ± 0.1921.42 ± 0.2122.20 ± 0.0721.85 ± 0.1621.93 ± 0.2222.24 ± 0.22
a=0a=021.96 ± 0.1322.15 ± 0.1921.96 ± 0.1821.88 ± 0.2322.03 ± 0.2121.99 ± 0.45
change+0.69+0.73−0.24+0.03+0.10−0.25

In the submitted version, the GELU, SiLU and Mish values at a=0a=0 and a=1a=1 came from seed-0 runs that had stopped after 10 to 37 epochs; these runs are retrained for 200 epochs. In Table 1, those cells had averaged the two completed seeds.

The two activations whose Jacobian falls (ELU, SELU) are the two whose corruption accuracy clearly improves; for GELU, SiLU and Mish, whose Jacobian changes by less than 10%, the change is within seed variation. The input-Jacobian penalty lowers the Jacobian more directly, which is why we penalize it for robustness (Section 6.5). The CIFAR-10-C half of Table 19 is still single-seed and mixed (SELU +0.72, ELU −0.49).

W3. ELU/SELU at a=0a=0 and ReLU. ELU, SELU and LeakyReLU at a=0a=0 are standardized ReLU; this is the purpose of the intervention, a controlled test of the leak rather than a proposed activation (Section 4). The gain from a=1a=1 to a=0a=0 measures the cost of the leak within each family. GELU, SiLU and Mish at a=0a=0 keep a nonlinear positive branch.

W3. Gated variants and ReLU on WRN-28-10. We do not claim that a gated activation beats ReLU. ReLU has no leak, so the leak account predicts that closing the negative branch brings an activation to the level of ReLU, not above it, and that at a=1a=1 the shortfall from ReLU grows with the leak. Table 13 shows both on WRN-28-10 (CIFAR-100). At a=1a=1, GELU, ELU and SELU trail ReLU (81.3281.32) by 2.62, 8.57 and 11.00 points, in the order of their leak (at a=1a=1, ELU and SELU also fit only 82 to 91% of the training set, Table 13). At a=0a=0, all three are within 0.51 points of ReLU (81.0481.04, 80.8180.81, 81.1781.17). On ResNet-18 (Table 1), the six activations at a=0a=0 (76.4676.46 to 76.7576.75) are likewise at the level of ReLU (76.3876.38).

Q2. ReLU reference and WRN-28-10 ELU. The ResNet-18 gating runs and their ReLU reference use the ResNet-18 training command in the code released by Zhang et al., without further tuning: SGD, learning rate 0.1, weight decay 10−410^{-4}, batch size 128, 10-epoch warm-up and cosine decay, 200 epochs (Table 23 and Appendix N.2). The correlation grid and all other backbones use their published configuration (weight decay 5×10−45\times10^{-4}, Table 23). The ReLU reference is ReLU without output standardization: 76.38±0.0776.38 \pm 0.07 over three seeds (one seed in the submitted version, as the other two had stopped at epochs 152 and 145 and are now retrained; standardized ReLU: 76.53±0.1076.53 \pm 0.10, Table 1 caption). On WRN-28-10, two of the three ELU runs at a=0a=0 in the submitted version had been stopped before completion (at epochs 6 and 20) on a shared machine and were recorded as diverged. All three seeds now train to 200 epochs: 80.81±0.3780.81 \pm 0.37, a gain of +8.06+8.06 over a=1a=1, the same behaviour as SELU. This is expected: SELU is a scaled ELU, so at a=0a=0 both are the same standardized ReLU (Table 13).

Q3. Validation versus test selection of aa. On CIFAR-100/ResNet-18, a=0.013a=0.013 was selected on a held-out 10% validation split (12 trials with GELU) and used unchanged for every activation. The earlier best-aa rows of Table 14 (Table 7 in the submitted version) came from a test-set grid. The same validation search on CIFAR-10/ResNet-18 selects a=0.013a=0.013 again. Revised Table 14 (three seeds, last epoch):

Activationa=1a=1a=0a=0a=0.013a=0.013 (validation)
GELU94.72 ± 0.2394.83 ± 0.1094.87 ± 0.05
SiLU94.52 ± 0.1094.92 ± 0.0594.79 ± 0.06
Mish94.54 ± 0.1294.96 ± 0.1094.71 ± 0.18
ELU93.74 ± 0.1494.71 ± 0.1694.77 ± 0.13
SELU93.53 ± 0.1294.86 ± 0.1394.76 ± 0.23

With validation selection, reducing the leak keeps its full gain (ELU +1.03, SELU +1.23), and the small leak is within 0.25 points of a=0a=0 for every activation. Appendix O lists the source (validation or test-set grid) of every selected aa. Table 14 uses the last epoch. The other accuracy tables report the best epoch, which in final runs (trained on the full training set, with no validation split) is the epoch with the highest test accuracy; the submitted version called these best-validation checkpoints, and Appendix O now states this. Table 25 repeats Table 1 at the last epoch: the mean gain of a=0.013a=0.013 over a=1a=1 is +0.77+0.77 (best epoch: +0.72+0.72).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions

We thank all three reviewers for their detailed and constructive reviews. The revised PDF shows all changes in blue; table and figure numbers here and in the replies refer to it. Tables 6 to 12, 16 and 25 and Figure 5 are new, so the tables after Table 5 are renumbered; main-text section numbers are unchanged.

Note on the correlation grid. While preparing the responses, we checked every model of the correlation grid (Figure 1, Table 5; 73 models). Three had stopped before the 200-epoch schedule under an early-stopping rule: ReLU and SELU on CIFAR-10/ResNet-34 (epochs 11 and 84) and ReLU on CIFAR-100/ResNet-18 (epoch 17). In addition, for activations implemented in place, the leak had been read after the activation overwrote its input, and Mish was recorded as zero. We regret that these models were incomplete in the submitted version. The three models are retrained for the full 200 epochs, the leak is read on the pre-activation (checked against automatic differentiation), and Figure 1, Table 5 and every table built on the grid are recomputed (Appendix O). The abstract numbers from Figure 1 and Table 1 are updated.

1. Statistical strength of the correlation analysis (MfFr, Hd4D)

  • Table 5 now reports all 13 measures of Figure 1 on all 11 dataset-architecture settings (73 models), with their mean.
  • Grid-level mean Kendall τ\tau with bootstrap intervals and a permutation test of the mean (Section 6.2); per setting and for all 13 measures, Spearman's ρ\rho and bootstrap intervals (new Table 11) and exact pp-values (new Table 12).
  • Four generalization targets: the accuracy gap (primary, as in the submitted version), the loss gap of Zhang et al., and both gaps with training accuracy and loss recomputed at inference, without data augmentation (new Table 6, all 13 measures), and the loss-gap grid for all 11 settings (new Figure 5). Rényi sharpness is significant under no target; the leak and the input Jacobian under all four.
  • The CIFAR-100/ResNet-18 grid repeated with three seeds per activation, and under SAM and AdamW (new Table 7).

2. Weight-space measures and the optimizer axis (MfFr, Hd4D, ucgy)

  • Layer-wise Rényi sharpness on the 63 models of the nine settings with all seven activations (new Table 10). It is weak on the activation axis, like the global measure.
  • Rényi sharpness, trace, parameter norm, the leak and the input Jacobian on the 72 six-optimizer models of Table 21 (Table 13 in the submitted version), with the same estimators (new Table 8). Rényi sharpness (α=1.5\alpha=1.5) recovers a strong correlation there (mean τ\tau +0.67), so the difference lies in the axis of variation.
  • One predictor across both axes, fit by leave-one-setting-out on 15 settings (11 activation settings, 4 optimizer settings; 145 models), with single measures, combinations and weights (new Table 9).

3. Mechanism (MfFr, ucgy)

  • Section 3 states the mechanism at the level of the trained network, with new Table 16 (Appendix G.1) decomposing the input Jacobian layer by layer (two seeds, 38 models): gating raises the activation-layer gain, the learned layers lower theirs by more, and the Jacobian falls by 33 to 52% for ELU and SELU.
  • An "Interpretation" paragraph in the Discussion (Section 7).
  • The rationale for penalizing the leak rather than the coefficient aa, in Section 5 after Eq. (5).

4. Interventions and robustness (MfFr, ucgy, Hd4D)

  • Table 1: every cell from three complete seeds, with aa selected on a held-out validation split, and the ReLU reference's configuration. In the submitted version, the seed-0 GELU, SiLU and Mish runs at a=0a=0 and a=1a=1 had stopped after 10 to 37 epochs (those cells averaged the two completed seeds), and the ReLU reference had one seed; these are retrained.
  • Table 14 (Table 7 in the submitted version): a=0.013a=0.013 selected on a held-out validation split (as on CIFAR-100), three seeds; Appendix O lists the source of every selected aa.
  • Table 13 (Table 6 in the submitted version): WRN-28-10 ELU at a=0a=0 with three seeds, each trained for 200 epochs (+8.06+8.06 over a=1a=1).
  • Table 19 (Table 11 in the submitted version): CIFAR-100-C with three seeds per cell (seed 0 only before).
  • Section 4: the small-leak paragraph (ResNet-18 vs ResNet-20) is replaced with the validation-based account.

5. Clarity and protocol

  • Figure 1 caption: the five settings are those of Figure 3 of Zhang et al.; number of seeds and seed variation, with a pointer to Table 7.
  • Appendix O (protocol): correlation-grid training and leak readout, seeds, selection of aa, checkpoint rule (with no validation split in final runs, the best epoch is the one with the highest test accuracy, not best-validation as the submitted version said; new Table 25 repeats Table 1 at the last epoch), and corruption benchmarks.

We are happy to provide any further detail during the discussion.

AuthorsOpenReview ↗

Summary of the rebuttal

We thank Reviewer Hd4D for the follow-up discussion. For Reviewers MfFr and ucgy, we summarize below how the revision addresses their reviews and the main conclusions of the rebuttal. All changes are in blue in the revised PDF, and the detailed responses are under each review.

Reviewer MfFr

  • Loss gap. All 13 measures under four targets: accuracy and loss gaps, also with training accuracy and loss recomputed at inference (Table 6, Figure 5).
  • Layer-wise Rényi sharpness on 63 models (Table 10), and Rényi sharpness on the 72 six-optimizer models as a positive control (Table 8).
  • Strength of the leak. All 13 measures on all 11 settings (Table 5) with exact per-setting pp (Table 12); Section 6.2 now discusses trace and parameter norm.
  • Mechanism. A layer-by-layer decomposition of the input Jacobian on 38 trained networks (Table 16).
  • Seeds and protocol. CIFAR-100-C with three seeds (Table 19), the ReLU reference and WRN-28-10 ELU with three seeds, and aa selected on a validation split (Table 14, Appendix O).

Reviewer ucgy

  • ResNet-18 vs ResNet-20. Re-examined with validation-selected aa and three seeds (Section 4, Tables 14 and 18).
  • Combined predictor. One predictor across both axes, scored on held-out settings (Table 9).
  • Interpretation. A new paragraph in the Discussion (Section 7), and the rationale for penalizing the leak rather than aa (Section 5).

Key conclusions

  1. Which geometry predicts the gap depends on what varies. When the activation varies (11 settings, 73 models), Rényi sharpness is not significant under any of the four targets, while the leak (mean τ\tau +0.49, p<0.001p<0.001) and the input Jacobian (+0.36, p<0.001p<0.001) are significant under all four. When the optimizer varies (72 models), the same estimator tracks the gap strongly (+0.67, p<0.001p<0.001).
  2. Input-space measures work on both axes. In one predictor fit across 15 settings (145 models), the input Jacobian and the leak receive most of the weight (0.55 and 0.32); parameter norm, which reverses sign between the axes, receives 0.01. The same holds with the Hessian trace or parameter norm in place of Rényi sharpness: the input Jacobian carries the predictor, and weight-space measures combined without it do not predict across both axes (reply to Reviewer Hd4D).
  3. The leak acts on generalization. Reducing it with everything else fixed raises clean accuracy (CIFAR-100: ELU +1.53, SELU +2.45; three seeds), accuracy follows the leak at a fixed slope jump (r=−0.96r=-0.96), and closing the negative branch lowers the input Jacobian by 33 to 52% for ELU and SELU.
  4. The results hold across seeds and optimizers. On three seeds and under SAM, the leak is significant on CIFAR-100/ResNet-18; under SAM and AdamW, Rényi sharpness does not track the activation-induced gap.

We would be happy to provide any further clarification or answer any remaining questions from Reviewers MfFr and ucgy before the end of the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.