- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Piecewise Bernstein Networks (PBN), a family of learnable activation functions based on continuous piecewise Bernstein polynomials. The method can be used as a drop-in replacement for standard activations in MLPs, ResNets, and transformers. The authors provide theoretical results on exact representation of ReLU, LeakyReLU, and PReLU, as well as approximation of continuous activation functions. Experiments across function fitting, tabular classification, and vision tasks show improvements over ReLU in several settings. However, the contribution of the piecewise formulation over existing Bernstein activations remains unclear.
Critical strengths
-
PBN can be integrated into standard network architectures without changing their structure. The theoretical analysis also provides useful guarantees on continuity and approximation.
-
The paper evaluates PBN across several architectures and tasks, with promising results on ImageNet and vision transformers. The supplementary material provides extensive experimental details and analyses of accuracy, training time, and memory.
Critical weaknesses
-
The (K=0) variant is the existing DeepBern-Nets activation, and learnable spline and piecewise-polynomial activations have also been studied in prior work. Moreover, many of the strongest results are obtained with (K=0), while the benefit of the proposed piecewise formulation is less clear. The related work should better establish what is new compared with these methods.
-
The exact representation theorem assumes linear extrapolation outside the activation domain, while most experiments use clamping.
-
In addition, the comparisons are not fully controlled, since initialization and weight-decay choices differ across methods, and the KAN baseline uses a fixed grid without updates.
-
The ImageNet results also rely on only two seeds. This is insufficient to calculate a standard deviation.
-
Finally, there are some internal inconsistencies, including the regression suite containing 53 targets in the setup but 52 in the reported analysis.
Questions for authors
1- What is the main technical contribution beyond existing Bernstein activations and prior learnable spline or piecewise-polynomial activations? In particular, which results demonstrate the specific benefit of the piecewise formulation?
2- How do the theoretical claims about exact representation and universality relate to the clamped activation used in most experiments?
3- Could the authors also clarify the discrepancy between the 53 targets listed in the setup and the 52 targets analyzed?
4- Could the authors provide more controlled comparisons using matched initialization and regularization, together with a stronger KAN baseline that enables grid updates? Additional ImageNet seeds would also help establish the robustness of the reported gains.
3 comments
We thank the reviewer for recognizing that PBN integrates into standard architectures, that its theory provides useful guarantees, and that the ImageNet and vision transformer results are promising.
Below, we summarize the reviewer's suggestions and our response, which has further strengthened our work. We believe the additional analysis addresses all of the reviewer's questions.
- Contribution (W1 and Q1). With PBN with dynamic , the piecewise form is the most accurate PBN form in function fitting, on the HIGGS MLP and on the FT-Transformer.
- Clamping (W2 and Q2). Theorem 3.1 now states the interval on which it holds.
- Controlled comparisons and KAN (W3 and Q4). The limitations now state the common recipe, and KAN was also trained with its native procedure.
- ImageNet seeds (W4). All ImageNet configurations now have three seeds.
- 53 against 52 targets (W5 and Q3). The setup and the Table 1 caption now explain it.
We appreciate the reviewer's feedback, which has helped us clarify the paper's contributions and strengthen its experimental support.
Thank you for the response and additional experiments. However, the rebuttal does not sufficiently address my main concerns regarding novelty, significance, and the fairness of the experimental comparisons. The additional contributions are interesting but do not substantially change my overall assessment. I therefore maintain my original recommendation of rejection.
Response to Reviewer 34JA
We thank the reviewer for the detailed evaluation. Numbers are from the table in our general response.
W1 and Q1. Novelty and empirical benefit of the piecewise form. The paper states that K=0 is the DeepBern-Nets activation (Khedr & Shoukry, 2024), originally designed for certification. Our contributions beyond it are the following. (i) The piecewise formulation of K+1 Bernstein polynomials of degree n with C0 joins, with the exact representation of ReLU, LeakyReLU, and PReLU (Theorem 3.1) and an approximation bound that improves as K increases (Corollary 3.2). (ii) Dynamic K, which increases the number of knots during training without altering the function, as the Bernstein form permits. (iii) Fused Triton GPU kernels, which evaluate the unit without storing intermediate basis tensors. On the HIGGS MLP, they make an epoch 5.8 times faster and halve the peak memory against the same network without them (Table 11 of the revised paper). This made larger networks possible. To our knowledge, this gives the first evaluation of Bernstein-basis activations in ImageNet ResNets, an FT-Transformer and a ViT, for both K=0 and K≥1.
Among previous piecewise and spline activations, adaptive piecewise-linear units are of degree 1. The cubic B-splines used in KAN are C2 at their knots and cannot represent a kink, and KAN replaces entire network layers rather than the activation. PBN supports any polynomial degree, uses a basis whose coefficients bound its output, ensures continuity at the cost of one shared coefficient per knot, and can increase K during training without changing its function. The revised introduction cites further piecewise-linear units (Zhou et al., 2021, Bohra et al., 2020).
PBN with dynamic K is the most accurate PBN form in function fitting at all tiers (0.34, 0.16, and 0.13×10−6 vs 1.34, 0.40, and 0.21 for K=0), on the HIGGS MLP (.8724 vs .8699), and on the FT-Transformer (.8724 vs .8707). The fixed K≥1 form is the most accurate PBN form on ResNet-18 CIFAR-100 (75.61 vs 75.09).
W2 and Q2. Exact Representation vs. Boundary Clamping. Theorem 3.1 and Corollary 3.2 are statements on [l,u] activation interval and hold for the clamped units there. Theorem 3.1 holds on the interval [l,u] under either boundary rule, since the unit equals ReLU for all inputs within [l,u]. Linear extension is needed only for equality outside the interval (Appendix A.3). Our experiments use [−5,5] PBN interval, and where batch normalization precedes the activation, only 0.23% of inputs fall outside it. The FT-Transformer lacks preceding normalization, so its units use the linear extension. With clamping instead, the AUC is .8710 against .8707 (K=0) and .8716 against .8717 (K≥1). In conclusion, both results hold where the units operate, and the boundary rule changes the results by at most .0003 AUC. Theorem 3.1 now states the interval explicitly.
W3 and Q4. Controlled Comparisons and KAN Grid Updates. Within a setting, all networks are trained with the same optimizer and the same weight decay on their weights. In the convolutional and transformer networks, weight decay applies only to convolution and linear weights, so all learnable activations (PBN, BerLU, GR-KAN) are exempt. In MLP settings, Bernstein coefficients are exempt like biases, whereas in KAN, Chebyshev, and Fourier networks the basis coefficients are the only weights and are therefore decayed. Regarding initialization, the PBN forms start differently, as noted in our limitations, but this only affects comparisons between PBN forms. Each baseline keeps the initialization of its reference implementation.
Regarding KAN grid updates, this comparison concerns the two MLP settings only. KAN cannot be placed as an activation in a ResNet or a transformer. For a fair comparison, every architecture in a setting uses the same standard recipe, so KAN ran with AdamW on a fixed grid and not with its own LBFGS procedure and grid updates. The revised limitations paragraph now states this. As requested, we trained KAN with its native procedure on the nine targets of the KAN paper. KAN was more accurate than PBN on six targets (xy, x/y, esinπx+y2, eJ0(20x)+y2 and two composite targets), and PBN was more accurate than KAN on three (J0(20x) and two multivariate targets).
W4. ImageNet Seeds. All ImageNet configurations now have three seeds. PBN (K=0) is 1.00 and 0.61 points above ReLU on ResNet-34 and ResNet-50.
W5 and Q3. Target Discrepancy (53 vs 52). All 53 targets were run. The 100-dimensional one cannot be constructed within the 103-parameter tier by KAN and Fourier. It is therefore dropped from every geometric mean, which uses the same 52 targets at all three tiers. However, the 100-dimensional target is included in Figure 21, which compares the activations on each target separately. The setup section and Table 1 caption now state this.