- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies the following question: instead of fixing an activation function and training the network weights, we take the network weights as fixed and train the activation function (where the function is constrained to come from a certain function space) but with some regularizer. There are a few different results. For example that a minimum exists or proving certain properties of the minimizer.
Critical strengths
I am not very familiar with optimizations over a space such as . Therefore it is possible that I missed some important big insight.
The paper appears to be a quite well-rounded study in this setting. If one accepts the premise, it does cover, depending on the exact setting, existence, structure of the minimizer, convergence rate, etc.
Critical weaknesses
Parts of the paper (e.g. Lemma 4) are used from [17] and no independent argument is provided here. I do not have access to [17] (my understanding is that it is a different paper by the same authors and currently under peer review as well). Maybe it would be better to have [17] publish and peer reviewed first if the current submission depends on it? Or to combine the two papers. It is unfortunate, and not necessarily the authors fault, but I find it difficult to evaluate this submission completely independently from [17] when it so much relies on [17].
I think the paper would also be improved by having more discussion on why anybody would want to do this kind of optimization (in particular while keeping the network weights fixed) and/or what can be learned from it.
No code is provided.
Questions for authors
See the second point in weaknesses in particular. Also, if we have a lot of training points and neurons, doesn't this significantly constrain the activation function? Possibly to the point where it necessarily has to an extremely close approximation of the original one?
Thank you for your feedback.
See general response bullet 1. The revision is self-contained now; Lemma 4 and the representation formula are proved in Appendix C.
See general response bullet 2 and Table 2 (reproduced under Reviewer 1's comment). What is learned: (i) the optimal activation for a trained network is a smooth spline whose knots are the network's own pre-activations, a fixed point rather than free parameters; (ii) piecewise-linearity arises as the minimizer of a worst-case sensitivity bound; (iii) the same theory holds with one activation per neuron (new Proposition 12); (iv) in practice, redesigning the activation is the right repair when the damage is a non-affine distortion of the activation (saturation, in our synthetic model), and a competitive one otherwise.
Now provided; see general response bullet 3.
A good question with two different answers, and the revision states both (new paragraph in Section 5).
In the extremal problem: yes. Values are pinned at every operating point, and as the operating set densifies the best attainable slope tends to that of the original activation. We measure the ratio as 1 to within 2×10−7 at realistic densities; a gap opens only when the knot set is coarsened to tens of knots. The paper reports this as the dense-limit prediction of Theorem 15, not as an improvement.
In the regularized problem: no. No value is pinned. The data enter only through the risk, and the regularization weight λ alone decides how far the optimum moves. For large λ, the optimum stays close to the original; at regularization strengths of the size selected in the repair experiment, it moves substantially (Fig. 1, an illustrative instance at λ=4×10−3, shows a correction of size 0.4), and what limits it is overfitting the calibration set, which is what the selection of λ controls.