Self-Consistent and Extremal Activation Functions: A Variational Theory of Activation Design

Which activation function should a trained network have? We pose activation design as a variational problem on the affine Sobolev space ${\sigma }_{b}ase+H^2(R)$–minimize empirical risk plus an $H^2$ penalty over $\sigma$, with weights held fixed–and characterize the solutions structurally. Minimizers exist, and every critical point satisfies the self-consistent representer equation ${\sigma }^{\star }={\sigma }_{b}ase-\frac{1}{\lambda }G\ast {\bar{\mu }}_{{\sigma }^{\star }}$, with $G$ the Green's function of $(1-{\partial }_u^2{)}^2$ and ${\bar{\mu }}_{{\sigma }^{\star }}$ the network's data-averaged perceptual measure: the optimal activation is a $C^2$ exponential spline whose knots are the pre-activation values the network itself produces under ${\sigma }^{\star }$–not parameters, but a fixed point. Above an explicit threshold, the problem is strongly convex on a ball containing every minimizer; the fixed point there is unique, and Picard iteration gradient descent at step $1/\lambda$ converges geometrically, at one backward pass per sample per iterate. An extremal strand explains piecewise-linearity: among all activations leaving every prediction of a trained network unchanged, the piecewise-linear interpolant minimizes the uniform first-order attack-surface bound, and under gradient fidelity the extremal has bang-bang curvature — a perfect spline in the sense of Favard and Karlin. Finally, parametric activation families (adaptive splines, Kolmogorov–Arnold edges) are Galerkin sections that $\Gamma$-converge to the full problem at the rate $O(h^{3/2})$. Every prediction is verified on trained MNIST networks over five seeds: the contraction exponent within $0.3\%$ of theory, knot self-consistency at grid resolution, the $h^{3/2}$ rate, and machine-precision behavioral equivalence of the piecewise-linear representative.