- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies activations formed by summing shifted copies of a shared scalar activation. It analyzes restricted slope increments under fixed coefficients and persistent ties between branches, including an AdamW normalization effect for unequal coefficient magnitudes. It proposes initializing one branch with coefficient one and additional branches with zero coefficients at distinct locations, preserving the pretrained function. Supporting results include spline expressivity and a computer-assisted Boolean lower bound. Experiments show improvements over a same-site K=1 control on four frozen ResNet-50 transfer tasks, with limited benefits on ViT and longer-budget Pets experiments.
Critical strengths
-
The paper provides concrete diagnostics of branch coincidence and distinguishes nominal branch count from the number of distinct learned knots. The explicit Boolean witness and exact-enumeration procedure also make the finite-domain lower bound auditable.
-
The empirical reporting includes consistent improvements over the selected K=1 control across four ResNet-50 datasets, together with neutral and negative results. The discussion of longer training budgets and larger K helps establish the limitations of the approach.
Critical weaknesses
-
APL already learns hinge locations and coefficients from a shared scalar input, and SPLASH already initializes the ReLU coefficient to one and the additional coefficients to zero. This paper uses trainable knots and channel-specific parameters, but does not clearly establish their benefit over these existing approaches. In the main (1,-1,1) example, the positive branches also remain tied under SGD; the AdamW-specific result concerns unequal coefficient magnitudes.
-
Section 4.3 admits a simple counterexample. With fixed coefficients (1,-1,1) and independent biases (d,d,0), where d is nonzero, we have f(z)=sigma(z+d)-sigma(z+d)+sigma(z)=sigma(z). The initial function is preserved, the positive branches are already separated, and the cancelling pair can separate through opposite nonzero bias gradients.
The experiments also change coefficient training and initialization together, so they do not isolate the effect of breaking the tie. Does K=1 learn its coefficient? Table 8 suggests otherwise, and Table 5's equal-parameter-count claim needs reconciliation with Table 3. Table 13 also provides no clear advantage over APL or KAN-style activations. These comparisons leave the source of the reported gains unresolved.
References
- Agostinelli et al. (2015): “Learning Activation Functions to Improve Deep Neural Networks.” ICLR Workshop. https://arxiv.org/abs/1412.6830
- Tavakoli et al. (2021): “SPLASH: Learnable activation functions for improving accuracy and adversarial robustness.” Neural Networks, 140, 1–12. https://doi.org/10.1016/j.neunet.2021.02.023
Questions for authors
-
What is the precise methodological advance over APL's learned hinge parameterization and SPLASH's ReLU-preserving zero-coefficient initialization? Which theoretical or experimental result establishes the importance of the remaining differences?
-
How does the cancellation-pair initialization with coefficients (1,-1,1) and independent biases (d,d,0) affect the claim in Section 4.3? How does this function-preserving fixed-coefficient baseline compare with nested under the same training protocol?
-
With all coefficients trainable and both initializations preserving the pretrained function, does initializing the added zero-coefficient branches at distinct knots outperform initializing them at tied knots? This comparison would directly test the claimed benefit of symmetry breaking without simultaneously changing coefficient freedom or initial function preservation.
-
Does the K=1 control learn its coefficient? Please reconcile the parameter counts in Tables 3, 5, and 8 and report whether the improvement persists against K=1 with both a learned coefficient and a learned bias. An additional comparison with fixed versus trainable knots in the nested model would help identify the source of the gain.
Thank you for the careful revision and the corrected framing. However, none of the eight primary contrasts is significant (nested vs. learned-amplitude K=1: +0.03/+0.56 with intervals spanning zero), and APL outperforms the method on Pets, CIFAR-100, and Food-101. With the remaining theory largely following from standard results, the revised paper no longer supports a practical contribution, so I will keep my score.