- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(−1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates an interesting and relatively unexplored question: which geometric quantity can predict generalization differences when the optimizer is fixed and the activation function changes? The author found that Rényi sharpness is a reliable generalization prediction metric under optimizer changes, but becomes inconsistent under activation function changes. As an alternative, the author proposes an activation level called negative branch leakage.
Critical strengths
1.This paper discusses an interesting problem and provides an intuitive solution for negative branch leakage. 2.The activation functions covered by the experiment are sufficient, and there are seven common activation functions.
Critical weaknesses
1.The small sample size problem in correlation analysis weakens the statistical reliability of the core claims. 2.The evidence provided in the paper regarding the causal leakage to generalization is insufficient.
Questions for authors
1.Can the paper provide experiments on other commonly used optimizers, such as SAM, Adam, etc.? 2.The Kendall coefficient is not sufficient to support the conclusion. Can more evidence be provided to support the conclusion?
2 comments
Response to Reviewer Hd4D (Part 2/2)
W2. Causal evidence linking the leak to generalization. The paper tests the leak with three interventions that target it with the architecture, optimizer and schedule fixed, plus a new mechanism measurement. On ResNet-18, training accuracy exceeds 99.9% in every run of Tables 1 and 14, so there a higher test accuracy is a smaller generalization gap.
- Gating within an activation (Eq. 4). The coefficient scales only the negative branch, with architecture, optimizer and schedule fixed. With selected on validation and three seeds, reducing the leak improves clean accuracy over the original activation for the high-leak activations on CIFAR-10 (ELU +1.03, SELU +1.23; Table 14) and on CIFAR-100 (ELU +1.53, SELU +2.45; Table 1). Closing the branch () also raises accuracy on WRN-28-10 (SELU +10.85, ELU +8.06, three seeds each; Table 13), where ELU and SELU at do not fit the training set (82 to 91%, Table 13), so there the gain includes better fitting.
- Leak versus slope jump (Table 2). A Softplus grid varies the leak at a fixed slope jump. Accuracy follows the leak ( over nine cells, three seeds each); a joint fit attributes the effect to the leak (), not to the jump.
- Training-time control (Eq. 5). Penalizing the leak during training, with the activation unchanged, improves clean accuracy under standard SGD in three of four architecture-dataset pairs (Table 21).
- New: where the effect acts. Layer-by-layer measurement on 38 trained networks (two seeds) shows that closing the branch lowers the input Jacobian by 33 to 52% for ELU and SELU through the learned layers of the network (Table 16, Appendix G.1).
Together these support a causal role of the leak; it is not the only factor (Appendix L).
Q1. Other optimizers (SAM, Adam). The paper already evaluates five sharpness-aware optimizers in addition to SGD: SAM, ASAM, RSAM, FSAM and SSAM (Table 21; three seeds, two in four cells; four architecture-dataset pairs), and repeats the gating intervention under RSAM (Table 18), where it reproduces the SGD pattern. For the correlation study, we trained the full activation axis (seven activations, CIFAR-100/ResNet-18) under SAM () and under AdamW (Adam with decoupled weight decay), and recomputed every measure (Kendall , accuracy gap; exact in parentheses):
| Optimizer | Rényi | Leak | Input Jacobian | Trace | Param. norm |
|---|---|---|---|---|---|
| SGD (mean of 3 seeds) | +0.37 | +0.90 | +0.75 | +0.71 | +0.75 |
| SAM | +0.52 (0.07) | +1.00 (<0.001) | +0.81 (0.005) | +0.81 | +0.81 |
| AdamW | −0.33 (0.88) | +0.05 (0.50) | +0.62 (0.035) | +0.71 | +0.14 |
Under SAM the SGD pattern holds and is slightly stronger. Under AdamW the activation changes the gap much less (the seven gaps span 2.1 points, against about 5 under SGD), so the ranking is less determined; the input Jacobian and trace remain significant, the leak does not (+0.05), and Rényi sharpness again does not track the gap. The finding, that Rényi sharpness does not track the activation-induced gap while an input-space measure does, therefore holds under all three optimizers. Conversely, when the optimizer is varied (the six optimizers of Table 21, 72 models), the same Rényi estimator () tracks the gap strongly (mean +0.67, ): the contrast is between axes of variation. Both are new tables in Appendix F (Tables 7 and 8).
Response to Reviewer Hd4D (Part 1/2)
We thank you for recognizing the problem as interesting and the activation coverage as sufficient. We address each point with new analyses and new training runs, all added to the revised PDF; table numbers refer to the revised version, and W/Q follow the numbering of your review.
Note on the correlation grid. While preparing this response, we checked every model of the correlation grid (Figure 1, Table 5; 73 models). Three had stopped before the 200-epoch schedule under an early-stopping rule: ReLU and SELU on CIFAR-10/ResNet-34 (epochs 11 and 84) and ReLU on CIFAR-100/ResNet-18 (epoch 17). In addition, for activations implemented in place, the leak had been read after the activation overwrote its input, and Mish was recorded as zero. We regret that these models were incomplete in the submitted version. The three models are retrained for the full 200 epochs, the leak is read on the pre-activation (checked against automatic differentiation), and Figure 1, Table 5 and every table built on the grid are recomputed (Appendix O). Table 5 now reports all 13 measures of Figure 1 on all 11 settings (a setting is one dataset and one architecture, trained once with each activation), including the measures that are as strong as or stronger than the leak. All statistics below are computed on this completed grid.
W1 and Q2. Sample size and evidence beyond Kendall's τ. Each setting has seven models (five for ViT-B/16 and ResNet-50), so we added evidence at four levels.
The cross-axis predictor of Table 9 is also scored over 145 models (15 settings). Beyond correlation, the central evidence is interventional (W2) and does not depend on rank statistics.