acceptodds
Under review as a conference paper at ICLR 2027

Self-Consistent and Extremal Activation Functions: A Variational Theory of Activation Design

Abstract

Which activation function should a trained network have? We pose activation design as a variational problem on the affine Sobolev space –minimize empirical risk plus an penalty over , with weights held fixed–and characterize the solutions structurally. Minimizers exist, and every critical point satisfies the self-consistent representer equation , with the Green's function of and the network's data-averaged perceptual measure: the optimal activation is a exponential spline whose knots are the pre-activation values the network itself produces under –not parameters, but a fixed point. Above an explicit threshold, the problem is strongly convex on a ball containing every minimizer; the fixed point there is unique, and Picard iteration gradient descent at step converges geometrically, at one backward pass per sample per iterate. An extremal strand explains piecewise-linearity: among all activations leaving every prediction of a trained network unchanged, the piecewise-linear interpolant minimizes the uniform first-order attack-surface bound, and under gradient fidelity the extremal has bang-bang curvature — a perfect spline in the sense of Favard and Karlin. Finally, parametric activation families (adaptive splines, Kolmogorov–Arnold edges) are Galerkin sections that -converge to the full problem at the rate . Every prediction is verified on trained MNIST networks over five seeds: the contraction exponent within of theory, knot self-consistency at grid resolution, the rate, and machine-precision behavioral equivalence of the piecewise-linear representative.

Primary area:
optimization
Keywords:
activation functions, representer theorems, calculus of variations, splines, perfect splines, adversarial robustness, Gamma-convergence, Kolmogorov--Arnold networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8FxqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the following question: instead of fixing an activation function and training the network weights, we take the network weights as fixed and train the activation function (where the function is constrained to come from a certain function space) but with some regularizer. There are a few different results. For example that a minimum exists or proving certain properties of the minimizer.

Critical strengths

I am not very familiar with optimizations over a space such as H2(R)H^2(\mathbb{R}). Therefore it is possible that I missed some important big insight.

The paper appears to be a quite well-rounded study in this setting. If one accepts the premise, it does cover, depending on the exact setting, existence, structure of the minimizer, convergence rate, etc.

Critical weaknesses

Parts of the paper (e.g. Lemma 4) are used from [17] and no independent argument is provided here. I do not have access to [17] (my understanding is that it is a different paper by the same authors and currently under peer review as well). Maybe it would be better to have [17] publish and peer reviewed first if the current submission depends on it? Or to combine the two papers. It is unfortunate, and not necessarily the authors fault, but I find it difficult to evaluate this submission completely independently from [17] when it so much relies on [17].

I think the paper would also be improved by having more discussion on why anybody would want to do this kind of optimization (in particular while keeping the network weights fixed) and/or what can be learned from it.

No code is provided.

Questions for authors

See the second point in weaknesses in particular. Also, if we have a lot of training points and neurons, doesn't this significantly constrain the activation function? Possibly to the point where it necessarily has to an extremely close approximation of the original one?

1 comment

Thank you for your feedback.

Parts of the paper (e.g. Lemma 4) are used from [17] and no independent argument is provided.

See general response bullet 1. The revision is self-contained now; Lemma 4 and the representation formula are proved in Appendix C.

More discussion on why anybody would want to do this, in particular while keeping the weights fixed, and what can be learned from it.

See general response bullet 2 and Table 2 (reproduced under Reviewer 1's comment). What is learned: (i) the optimal activation for a trained network is a smooth spline whose knots are the network's own pre-activations, a fixed point rather than free parameters; (ii) piecewise-linearity arises as the minimizer of a worst-case sensitivity bound; (iii) the same theory holds with one activation per neuron (new Proposition 12); (iv) in practice, redesigning the activation is the right repair when the damage is a non-affine distortion of the activation (saturation, in our synthetic model), and a competitive one otherwise.

No code is provided.

Now provided; see general response bullet 3.

If we have a lot of training points and neurons, doesn't this significantly constrain the activation function? Possibly to the point where it has to be an extremely close approximation of the original one?

A good question with two different answers, and the revision states both (new paragraph in Section 5).

In the extremal problem: yes. Values are pinned at every operating point, and as the operating set densifies the best attainable slope tends to that of the original activation. We measure the ratio as 1 to within 2×10−72 \times 10^{-7} at realistic densities; a gap opens only when the knot set is coarsened to tens of knots. The paper reports this as the dense-limit prediction of Theorem 15, not as an improvement.

In the regularized problem: no. No value is pinned. The data enter only through the risk, and the regularization weight λ\lambda alone decides how far the optimum moves. For large λ\lambda, the optimum stays close to the original; at regularization strengths of the size selected in the repair experiment, it moves substantially (Fig. 1, an illustrative instance at λ=4×10−3\lambda = 4 \times 10^{-3}, shows a correction of size 0.4), and what limits it is overfitting the calibration set, which is what the selection of λ\lambda controls.

Reviewer_M9PDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • Formulates activation design as a variational problem over σbase+H2(R)\sigma_{base}+H^2 (R), with network weights held fixed.
  • Characterizes critical-point corrections as exponential splines whose knots depend self-consistently on the network’s pre-activations, and provides uniqueness and convergence results for sufficiently strong regularization.
  • Separately studies extremal piecewise-linear activations and establishes an O(h3/2)O(h^{3/2}) Galerkin approximation rate, supported by MNIST experiments.
Critical strengths
  • The function-space formulation connects activation design, regularization, regularity, and optimization in an interesting way.
  • Self-consistent knots determined by the network’s own forward pass provide a clear structural distinction from externally parameterized spline knots.
Critical weaknesses
  • The regularity argument underlying Lemma 4 requires clarification. Lipschitz continuity of the gradient map on an H2H^2 ball, with moving pre-activation locations, does not follow immediately from the stated assumptions.
  • The experimental Lipschitz constant is estimated from sampled ratios and multiplied by a safety factor of three. This does not provide a certified upper bound or validate the theoretical threshold λ0λ_0.
Questions for authors
  • Can Lemma 4 be established using only H2H^2 control, particularly for evaluating σ\sigma' at moving pre-activation locations? Is stronger regularity, such as W2,∞W^{2,∞} required?
  • Assumption 1 requires σbase∈Cb1σ_{base}∈C_b^1, but GELU is unbounded. Can the theory be extended to a linear-growth assumption covering the experimental setting?
  • How conservative is a genuinely certified bound on LRL_R? Does meaningful predictive improvement remain at the corresponding regularization strength?
  • The manuscript does not appear to follow the official ICLR submission format and should be revised to comply with the prescribed template.
1 comment

Thank you for your careful reading. We respond point by point; all numbering refers to the revision.

The regularity argument underlying Lemma 4 requires clarification. Lipschitz continuity of the gradient map on an H2H^2 ball, with moving pre-activation locations, does not follow immediately from the stated assumptions.

The reviewer is right, and the two-sided claim in the submitted version was false as stated. For general s∈H2s \in H^2, s′s' is only 12\tfrac12-Hölder, so evaluating s′s' at two pre-activations a distance ϵ\epsilon apart gives O(ϵ)O(\sqrt{\epsilon}), not O(ϵ)O(\epsilon). The revision replaces the lemma with an anchored estimate: the Lipschitz bound holds for pairs (s1,s2)(s_1, s_2) in the ball when at least one of the two has bounded second derivative (∥s′′∥∞≤Λ\|s''\|_\infty \le \Lambda); the full proof is in Appendix C. This suffices for every downstream result, because the fixed-point map Φ=−Φ0/λ\Phi = -\Phi_0/\lambda carries the ball into the set of curvature-bounded functions (with Λ∝1/λ\Lambda \propto 1/\lambda), so every fixed point anchors its own contraction estimates. Theorem 9 (uniqueness) and Corollary 21 (the approximation rate) are restated accordingly: quadratic growth around the critical point (Theorem 9(ii)) rather than strong convexity on the whole ball.

Can Lemma 4 be established using only H2H^2 control? Is stronger regularity, such as C1,1C^{1,1}, required?

H2H^2 control of both points plus a curvature (C1,1C^{1,1}) bound on one of them. No two-sided estimate holds with H2H^2 alone, and the revision says so explicitly in a remark after the lemma. C1,1C^{1,1} regularity of the whole space is not needed.

Assumption 1 requires bounded σbase\sigma_{base}, but GELU is unbounded.

Correct; this was an error in the submitted version. Assumption 1(iii) now asks only that σbase′\sigma_{\mathrm{base}}' be bounded and Lipschitz — σbase\sigma_{\mathrm{base}} itself may be unbounded — which covers GELU, SiLU, tanh, and softplus, while excluding ReLU (not C1C^1). Lemma 3 (operating radius) is reproved under the slope bound alone, by a layer-by-layer induction using ∣σ(u)∣≤∣σ(0)∣+Lip(σ) ∣u∣|\sigma(u)| \le |\sigma(0)| + \mathrm{Lip}(\sigma)\,|u|.

The Lipschitz constant is estimated from sampled ratios and multiplied by a safety factor of three. This does not provide a certified upper bound.

Agreed, and the revision no longer presents it as one. The text now states that the two binding constants (L^\widehat{L} and M^Rˉ\widehat{M}_{\bar R}) are sampled, that both are lower bounds on the true suprema, and that every claim at λ=2λ^0\lambda = 2\widehat{\lambda}_0 is conditional on them; the E1/E3 measurements (geometric decay, monotone descent, a single limit cluster) are reported as the direct evidence that the conditions hold in practice.

How conservative is a genuinely certified bound on λ0\lambda_0? Does meaningful predictive improvement remain at the corresponding regularization strength?

We measured this (E9, new), and the answer is no. Bounding MRˉM_{\bar R} in closed form gives MRˉcert∈[231,270]M_{\bar R}^{\mathrm{cert}} \in [231, 270] across seeds, so the certified threshold is λcert≈250\lambda_{\mathrm{cert}} \approx 250, about a thousand times the sampled one. At that λ\lambda the solution moves only ∥s⋆∥H2≤4.1×10−5\|s^\star\|_{H^2} \le 4.1 \times 10^{-5} from the base activation, the design risk falls by a relative 6.1×10−56.1 \times 10^{-5} at most, and the test loss moves by at most 10−710^{-7}. Uniqueness is fully certified only where there is almost nothing left to optimize. The useful regularization strengths (those selected in E8) lie far below the threshold.

Because of this we added a result that covers the regime the experiments actually use. Proposition 11 (new): for every λ>0\lambda > 0, the gradient iteration with an Armijo line search, started from zero, decreases the objective monotonically, drives the H2H^2 gradient norm to zero, stays in a compact set, and every cluster point is a critical point and hence a self-consistent exponential spline. What is lost below the threshold is only uniqueness and global optimality, and the paper now says exactly that. The proof runs on the anchored Lemma 4: every iterate is a convex combination of zero and images of the fixed-point map, all curvature-bounded, so every estimate along the path is anchored.

The revision also uses the official ICLR template.

Reviewer_rk7KOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to optimize activation functions with frozen weights (and sensitivity penalties) and studies the solutions to this optimization problem. Despite trying to read the paper multiple times, I cannot follow it well enough to understand the contributions.

Critical strengths

The idea of optimizing activations (with frozen) weights with some sensitivity penalty is really interesting and could have wide applications if e.g., the activation can be optimized for quantization or some other desired property.

Critical weaknesses

The writing is hard to follow. The motivation is unclear. Several of the core results seem to rely on a citation that is unpublished and not available to read, so the correctness cannot be verified.

Questions for authors

Can you please provide a motivation for the optimization problems you pose in the paper? And can you please provide the reference containing the core results?

1 comment

We thank you for your feedback.

The writing is hard to follow. The motivation is unclear. Can you please provide a motivation for the optimization problems you pose in the paper?

The introduction now opens with the problem in plain terms, explains why the weights are frozen, and states what the theory is for (see general response 2). In short: after training, the weights are often the one thing that cannot change (the model is quantized, ported to hardware, or audited, and retraining is unavailable), while the activation is a single scalar function and the cheapest object to replace. Table 1 is shortened to headline numbers, and the main text develops two results instead of four.

Several of the core results seem to rely on a citation that is unpublished and not available to read, so the correctness cannot be verified. Can you please provide the reference containing the core results?

The core results no longer rely on an unpublished citation. The result in question is now stated and proved in full in the paper (Appendix C; see general response 1), so there is no outside reference to provide: the revision cites no unpublished work.

The idea could have wide applications if e.g., the activation can be optimized for quantization or some other desired property.

We tried this. We ran it, together with two kinds of damage to the activation itself, on five seeds each of MNIST and CIFAR-10 networks (E8, Table 2). All repairs use the same 1000 calibration samples; every hyperparameter is selected on 2000 further samples; neither set was used in gradient updates of the networks (both come from the split held out for early stopping); the test set is used only to report. Mean test loss (lower is better):

MNIST (undamaged 0.053)

Repair4-bit weights3-bit weightsaffine mismatchsaturation
none0.0840.1350.0810.161
bias tuning0.0710.1030.0710.135
scale-and-bias tuning0.0760.1050.0710.090
shared designed activation0.0760.1210.0770.059
per-neuron designed activation0.0680.0980.0690.071
bias correction (needs undamaged network)0.0580.0910.0700.198
bias correction + per-neuron design0.0580.0880.0680.071

CIFAR-10 (undamaged 1.45)

Repair4-bit weights3-bit weightsaffine mismatchsaturation
none1.481.691.572.15
bias tuning1.471.571.491.58
scale-and-bias tuning1.451.531.441.61
shared designed activation1.431.551.501.47
per-neuron designed activation1.451.571.491.53
bias correction (needs undamaged network)1.481.621.552.38
bias correction + per-neuron design1.451.521.491.63

So on the quantization question specifically: activation design works and is competitive, but it is not the best tool at the moment. On MNIST, the per-neuron design is the best method that uses only calibration data, and analytic bias correction (which also uses the undamaged network) is better still at both 4 and 3 bits; on CIFAR-10, the shared design has the lowest loss at 4 bits and scale-and-bias tuning has the lower loss at 3 bits. Adding the per-neuron design after bias correction lowers the loss in 37 of 40 runs.

We state the limits of this comparison in the paper: the networks are MLPs, and the tuning baselines received a small hyperparameter search, so a stronger tuning protocol might narrow the gaps further. Whether activation design can become the preferred tool for quantization is open.

Where the method clearly wins is damage to the nonlinearity (saturation), which weight-space adjustments cannot fully undo: the designed activation has a lower test loss than every weight-space repair in all 10 runs (five seeds on each dataset). The saturation model is synthetic, and we say so in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

General response (all reviewers)

We thank the reviewers. Three concerns were shared, and we agree with all three.

1. The paper depended on an unpublished reference.

Fixed. The revision cites no unpublished work of ours. Lemma 4 is stated and proved in full in Appendix C, together with the derivation of the perceptual-measure representation it relies on. All 19 references are publicly available work by other authors.

2. The motivation for optimizing the activation with frozen weights was unclear.

The introduction is rewritten around this question. After training, the weights are often the one thing that cannot change (the model is quantized, ported to hardware, or audited, and retraining is expensive or unavailable), while the activation is a single scalar function and the cheapest object to replace. Freezing the weights is also what makes the problem tractable: the network then sees its activation only through finitely many pre-activation values, which is why theorems rather than search can describe the optimum.

The revision also answers the question empirically, which the submitted version did not. A new experiment, E8 (Table 2), damages trained networks and repairs them without retraining, against tuned weight-space baselines, on MNIST and CIFAR-10. In short:

  • When the damage is to the activation itself (each neuron's nonlinearity saturates at its own level), redesigning the activation recovers 94% (MNIST) and 98% (CIFAR-10) of the lost test loss and beats every weight-space repair in all 10 runs. On CIFAR-10, accuracy goes from 21.8% to 48.2% (undamaged: 51.6%); the best baseline reaches 43.9%.
  • When the damage is to the weights (quantization) or is affine, the method is competitive but not dominant; details are under Reviewer 1.

To make the paper easier to follow, we also (i) added Fig. 1, which shows the optimal activation, its correction, and the pre-activations where its knots sit; (ii) added a plain-language description of the main equation; and (iii) moved two secondary strands (the bang-bang curvature theorem and the Galerkin hierarchy) to Appendix B, so the main text develops one variational problem and one extremal problem.

3. No code was provided.

The supplement now contains all experiment code, the trained checkpoints for every seed, per-run configurations, and raw records. Every number in the paper is reproducible from those artifacts.

We have tried to state the limits as plainly as the results: the networks are MLPs, the damage models are synthetic, the method does not dominate on weight damage, the designs are computed on a finite kernel grid with a quasi-Newton solver, and the fully certified uniqueness threshold is too large to be useful.

Note on AI use. We used an AI assistant to edit these responses for clarity and concision. The content, claims, and numbers are our own, and we have checked them against the revised manuscript.

AuthorsOpenReview ↗

On AI Usage

We cannot update the submission; before we did not use AI for anything but to help streamline the process, we used AI to clean up the paper and ensure it is clearer.

Here is the official AI usage statement:

In this work, we used AI tools (Claude, Anthropic) extensively. The AI system wrote some of the experimental code, drafted and revised some parts of the manuscript, and acted as an adversarial reviewer of successive drafts. It also contributed to the mathematics, such as reviewing proofs and suggesting theorems. The problem formulation and the main theorems are the authors’. We have not used generative AI tools to select which experimental outcomes to report: every outcome of every run is reported, including the unfavorable ones. All AI-assisted work was reviewed: the proofs were checked by the authors, and the AI-generated code was validated by reproducing every reported number from the released artifacts. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI.

Additionally, rebuttal comments were checked by AI tools to ensure clarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.