acceptodds
Under review as a conference paper at ICLR 2027

Piecewise Bernstein Networks: Scalable, Compatible and Universal Learnable Activations

Abstract

Learnable activation functions (like Kolmogorov-Arnold networks KAN, Polynomials, and Rational) increase model expressivity by enabling individual neurons to adapt their non-linear shapes to data. However, three critical obstacles prevent them from replacing fixed activations: a lack of scalability to large, complex architectures such as Vision Transformers, a lack of architectural compatibility, and limited expressivity in existing flexible formulations. We introduce piecewise Bernstein networks (PBN), a family of learnable activations formed by Bernstein polynomials of degree joined continuously at interior knots. We prove that PBN neurons can exactly reproduce foundational activations like ReLU, LeakyReLU, and PReLU as special cases and can approximate any other continuous activation function, rendering them into universal neurons. Against ten other activations on 13 benchmarks and in six settings, spanning function fitting, tabular MLPs, ResNets on CIFAR and ImageNet, an FT-Transformer and a vision transformer, PBN ranks first on eight benchmarks and second on the other five, and it is never behind the best fixed activation beyond seed noise. It raises ImageNet top-1 over ReLU by 1.06 points with ResNet-34 and 0.73 with ResNet-50, and the accuracy of a vision transformer by 1.0 and 2.0 points on CIFAR-10 and CIFAR-100, at 1.4 to 3.1 times ReLU’s epoch time. On every benchmark, its validation-selected configuration reaches ReLU’s final validation value in fewer epochs than ReLU.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
activation functions, learnable activations, Bernstein polynomials, polynomial networks, Kolmogorov-Arnold networks, approximation theory, vision transformers, empirical comparison

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_34JAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Piecewise Bernstein Networks (PBN), a family of learnable activation functions based on continuous piecewise Bernstein polynomials. The method can be used as a drop-in replacement for standard activations in MLPs, ResNets, and transformers. The authors provide theoretical results on exact representation of ReLU, LeakyReLU, and PReLU, as well as approximation of continuous activation functions. Experiments across function fitting, tabular classification, and vision tasks show improvements over ReLU in several settings. However, the contribution of the piecewise formulation over existing Bernstein activations remains unclear.

Critical strengths
  • PBN can be integrated into standard network architectures without changing their structure. The theoretical analysis also provides useful guarantees on continuity and approximation.

  • The paper evaluates PBN across several architectures and tasks, with promising results on ImageNet and vision transformers. The supplementary material provides extensive experimental details and analyses of accuracy, training time, and memory.

Critical weaknesses
  • The (K=0) variant is the existing DeepBern-Nets activation, and learnable spline and piecewise-polynomial activations have also been studied in prior work. Moreover, many of the strongest results are obtained with (K=0), while the benefit of the proposed piecewise formulation is less clear. The related work should better establish what is new compared with these methods.

  • The exact representation theorem assumes linear extrapolation outside the activation domain, while most experiments use clamping.

  • In addition, the comparisons are not fully controlled, since initialization and weight-decay choices differ across methods, and the KAN baseline uses a fixed grid without updates.

  • The ImageNet results also rely on only two seeds. This is insufficient to calculate a standard deviation.

  • Finally, there are some internal inconsistencies, including the regression suite containing 53 targets in the setup but 52 in the reported analysis.

Questions for authors

1- What is the main technical contribution beyond existing Bernstein activations and prior learnable spline or piecewise-polynomial activations? In particular, which results demonstrate the specific benefit of the piecewise formulation?

2- How do the theoretical claims about exact representation and universality relate to the clamped activation used in most experiments?

3- Could the authors also clarify the discrepancy between the 53 targets listed in the setup and the 52 targets analyzed?

4- Could the authors provide more controlled comparisons using matched initialization and regularization, together with a stronger KAN baseline that enables grid updates? Additional ImageNet seeds would also help establish the robustness of the reported gains.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 34JA

We thank the reviewer for the detailed evaluation. Numbers are from the table in our general response.

W1 and Q1. Novelty and empirical benefit of the piecewise form. The paper states that K=0K=0 is the DeepBern-Nets activation (Khedr & Shoukry, 2024), originally designed for certification. Our contributions beyond it are the following. (i) The piecewise formulation of K+1K+1 Bernstein polynomials of degree nn with C0C^{0} joins, with the exact representation of ReLU, LeakyReLU, and PReLU (Theorem 3.1) and an approximation bound that improves as KK increases (Corollary 3.2). (ii) Dynamic KK, which increases the number of knots during training without altering the function, as the Bernstein form permits. (iii) Fused Triton GPU kernels, which evaluate the unit without storing intermediate basis tensors. On the HIGGS MLP, they make an epoch 5.8 times faster and halve the peak memory against the same network without them (Table 11 of the revised paper). This made larger networks possible. To our knowledge, this gives the first evaluation of Bernstein-basis activations in ImageNet ResNets, an FT-Transformer and a ViT, for both K=0K=0 and K≥1K\ge1.

Among previous piecewise and spline activations, adaptive piecewise-linear units are of degree 1. The cubic B-splines used in KAN are C2C^{2} at their knots and cannot represent a kink, and KAN replaces entire network layers rather than the activation. PBN supports any polynomial degree, uses a basis whose coefficients bound its output, ensures continuity at the cost of one shared coefficient per knot, and can increase KK during training without changing its function. The revised introduction cites further piecewise-linear units (Zhou et al., 2021, Bohra et al., 2020).

PBN with dynamic KK is the most accurate PBN form in function fitting at all tiers (0.34, 0.16, and 0.13×10−60.13\times10^{-6} vs 1.34, 0.40, and 0.21 for K=0K=0), on the HIGGS MLP (.8724 vs .8699), and on the FT-Transformer (.8724 vs .8707). The fixed K≥1K\ge1 form is the most accurate PBN form on ResNet-18 CIFAR-100 (75.61 vs 75.09).

W2 and Q2. Exact Representation vs. Boundary Clamping. Theorem 3.1 and Corollary 3.2 are statements on [l,u][l, u] activation interval and hold for the clamped units there. Theorem 3.1 holds on the interval [l,u][l, u] under either boundary rule, since the unit equals ReLU for all inputs within [l,u][l, u]. Linear extension is needed only for equality outside the interval (Appendix A.3). Our experiments use [−5,5][-5, 5] PBN interval, and where batch normalization precedes the activation, only 0.23% of inputs fall outside it. The FT-Transformer lacks preceding normalization, so its units use the linear extension. With clamping instead, the AUC is .8710 against .8707 (K=0K=0) and .8716 against .8717 (K≥1K\ge1). In conclusion, both results hold where the units operate, and the boundary rule changes the results by at most .0003 AUC. Theorem 3.1 now states the interval explicitly.

W3 and Q4. Controlled Comparisons and KAN Grid Updates. Within a setting, all networks are trained with the same optimizer and the same weight decay on their weights. In the convolutional and transformer networks, weight decay applies only to convolution and linear weights, so all learnable activations (PBN, BerLU, GR-KAN) are exempt. In MLP settings, Bernstein coefficients are exempt like biases, whereas in KAN, Chebyshev, and Fourier networks the basis coefficients are the only weights and are therefore decayed. Regarding initialization, the PBN forms start differently, as noted in our limitations, but this only affects comparisons between PBN forms. Each baseline keeps the initialization of its reference implementation.

Regarding KAN grid updates, this comparison concerns the two MLP settings only. KAN cannot be placed as an activation in a ResNet or a transformer. For a fair comparison, every architecture in a setting uses the same standard recipe, so KAN ran with AdamW on a fixed grid and not with its own LBFGS procedure and grid updates. The revised limitations paragraph now states this. As requested, we trained KAN with its native procedure on the nine targets of the KAN paper. KAN was more accurate than PBN on six targets (xyxy, x/yx/y, esin⁡πx+y2e^{\sin\pi x+y^2}, eJ0(20x)+y2e^{J_0(20x)+y^2} and two composite targets), and PBN was more accurate than KAN on three (J0(20x)J_0(20x) and two multivariate targets).

W4. ImageNet Seeds. All ImageNet configurations now have three seeds. PBN (K=0K=0) is 1.00 and 0.61 points above ReLU on ResNet-34 and ResNet-50.

W5 and Q3. Target Discrepancy (53 vs 52). All 53 targets were run. The 100-dimensional one cannot be constructed within the 10310^{3}-parameter tier by KAN and Fourier. It is therefore dropped from every geometric mean, which uses the same 52 targets at all three tiers. However, the 100-dimensional target is included in Figure 21, which compares the activations on each target separately. The setup section and Table 1 caption now state this.

We thank the reviewer for recognizing that PBN integrates into standard architectures, that its theory provides useful guarantees, and that the ImageNet and vision transformer results are promising.

Below, we summarize the reviewer's suggestions and our response, which has further strengthened our work. We believe the additional analysis addresses all of the reviewer's questions.

  • Contribution (W1 and Q1). With PBN with dynamic KK, the piecewise form is the most accurate PBN form in function fitting, on the HIGGS MLP and on the FT-Transformer.
  • Clamping (W2 and Q2). Theorem 3.1 now states the interval on which it holds.
  • Controlled comparisons and KAN (W3 and Q4). The limitations now state the common recipe, and KAN was also trained with its native procedure.
  • ImageNet seeds (W4). All ImageNet configurations now have three seeds.
  • 53 against 52 targets (W5 and Q3). The setup and the Table 1 caption now explain it.

We appreciate the reviewer's feedback, which has helped us clarify the paper's contributions and strengthen its experimental support.

Thank you for the response and additional experiments. However, the rebuttal does not sufficiently address my main concerns regarding novelty, significance, and the fairness of the experimental comparisons. The additional contributions are interesting but do not substantially change my overall assessment. I therefore maintain my original recommendation of rejection.

Reviewer_W2kxOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Piecewise Bernstein Networks (PBNs), a family of learnable activations based on Bernstein polynomials. A proof of universal approximation is provided, in addition to a performance comparison against several benchmark models on a suite of tasks. The empirical results indicate that PBNs are a promising avenue to improve expressivity and scalability, with a major limitation being increased wall-clock training time.

Critical strengths

The idea proposed is novel and has major advantages of scalability and compatibility. The exact recovery of piecewise linear baselines allows this method to recover the standard activations while expanding the representative capacity beyond their limits.

The empirical results included in the paper were on a diverse range of tasks and performance comparisons were made with a number of baselines. Additionally, the figures were high-quality.

The transparency of providing cost analysis of the proposed method showing the tradeoff between wall-clock time and performance improvements is appreciated.

Critical weaknesses

The authors did acknowledge this weakness in Section 6, but I am concerned about the small number of seeds used. Further, there was only a single configuration used for ImageNet with K≥1K \geq 1.

The argument that PBNs can amortize the computational cost by removing the need for trial-and-error activation search is not the strongest since there is still a hyperparameter search for K and n and the selected hyperparameters vary drastically by task.

Questions for authors

Given the small number of seeds used and the single configuration for ImageNet, will more statistically rigorous assessments (eg. standard deviations over more runs) be provided in the camera-ready version?

What is the sensitivity to hyperparameters K and n? If an ML practitioner were to use some default settings without grid searching, how much of the performance benefit is retained?

3 comments
AuthorsOpenReview ↗

Response to Reviewer W2kx

We thank the reviewer for the constructive assessment. The figures discussed below correspond to the updated table in our general response.

Seeds and ImageNet Evaluation As the reviewer suggested, we ran one more seed for each of the 14 ImageNet configurations (seven per depth), so every one of them now has three seeds (general response, item 1). We also ran PBN with dynamic KK (n=3,K→3n=3, K\rightarrow3) with one seed at both depths (general response, item 2). In the camera-ready version we can provide further configurations of PBN (K≥1K\ge1) and of PBN with dynamic KK over three seeds. The added seeds leave the conclusions unchanged. PBN (K=0K=0) remains the most accurate three-seed configuration, 1.00 points above ReLU on ResNet-34 and 0.61 on ResNet-50, and the two submitted PBN forms remain the two most accurate activations at both depths. K=0K=0 with n=3n=3 is the best configuration on ResNet-50 and within 0.06 points of the best on ResNet-34, while n=9n=9 stays at ReLU's level on ResNet-50 (77.02 against 77.05). The single seed of dynamic KK reaches 75.07 and 77.76, above the three-seed median of every other configuration. All other benchmarks already used three seeds, and our appendices detail the results for every configuration.

Simplifying the Hyperparameter Search for KK and nn The reviewer correctly notes that our submission relied on grid-searching both KK and nn. Dynamic KK and the results below allow us to recommend a default setting.

Regarding KK. With dynamic KK , the user does not need to choose the number of knots. Only an upper bound K‾\overline{K} is given, and KK increases during training up to it (general response, item 2). We recommend K‾=3\overline{K}=3 as the default. The table below shows that this default, used without any search over KK or nn, retains most of the gain in our benchmarks.

Regarding nn. PBN with dynamic KK exceeded ReLU's selected configuration in every benchmark column for all tested degrees (3, 5, and 9), with one tie (n=5n=5 on ResNet-18 CIFAR-10, yielding 94.54 against 94.55). The spread across degrees is small, namely 0.1 to 0.5 points on ResNet-18 and ViT, .0007 AUC on the FT-Transformer, and at most .003 on the HIGGS MLP. Function fitting showed the most sensitivity, with n=9n=9 reducing errors by a factor of 1.6 to 2.1 compared to n=3n=3. Lower degrees also provide computational savings. On the ViT with K=0K=0, an epoch at n=3n=3 requires 1.56 times ReLU's cost compared to 1.72 at n=9n=9.

Recommended Default Settings Without Grid Search We recommend dynamic KK with K‾=3\overline{K}=3, with n=5n=5 in MLPs and transformers and n=3n=3 in convolutional networks. The table below compares the n=5n=5 default with ReLU's validation-selected configuration and with the validation-selected PBN configuration, in the columns where it was run. (Network depth and learning rate are chosen on validation, as for ReLU.)

FF 10310^{3}FF 10410^{4}FF 10510^{5}HIGGS 10510^{5}FT-TViT C10
ReLU, selected522215.8634.870492.35
PBN, dynamic KK, n=5n=5, K‾=3\overline{K}=30.400.210.19.8719.872493.82
PBN, dynamic KK, selected0.340.160.13.8724.872493.87

This default configuration retains 100% of the PBN accuracy gains over ReLU on the FT-Transformer, 97% on ViT CIFAR-10, and 95% on the HIGGS MLP. In function fitting, its error is 1.2 to 1.4 times that of the selected PBN and 80 to 130 times lower than ReLU's error. In convolutional networks, n=3n=3 with K‾=3\overline{K}=3 is the configuration run on ImageNet (75.07 and 77.76 with one seed, against 73.96 and 77.05 for ReLU). On ResNet-18, where the upper bound was 7, n=3n=3 is the most accurate degree (94.80 and 75.58, against 94.55 and 74.88 for ReLU) and the three degrees are within 0.3 points of one another. In a parameter-matched MLP, a higher degree with more pieces leaves less width, which is why we recommend n=5n=5 and not a higher degree for MLPs.

The Amortisation Argument We agree with the reviewer's critique regarding total end-to-end compute, for which we have no measurement, and have removed the claim of a net reduction from the paper. The results support a narrower statement. PBN is first or second in every benchmark column, whereas the best fixed activation changes with the setting (ReLU, GELU, or ReGLU). With dynamic KK and our recommended default, most of the gain is obtained in our benchmarks without a grid search over KK or nn.

I thank the authors for the constructive rebuttal. The additional experiments address my main concerns.

Firstly, adding a third seed to the original ImageNet configurations improves the empirical evidence. Additionally, the hyperparameter sensitivity analysis and the recommended default settings make the method more practical to adopt without an exhaustive grid search. The introduction of the dynamic KK variant is a clever solution that appears to address the concern raised by the other reviewers on the K=0K=0 dominance.

However, some minor limitations lead to my maintenance of a 'Weak Accept' score. Specifically, the ImageNet results for the new dynamic KK variant are still based on a single seed. While I acknowledge the time constraints of the rebuttal period and the authors' commitment to adding these to the camera-ready version, further multi-seed evaluation of configurations at scale is necessary to fully support for the piecewise formulation. Additionally, the computational amortization argument is nuanced given the per-epoch overhead, as conceded by the authors.

Overall, I thank the authors for the rebuttal and I maintain my recommendation for acceptance.

We would like to thank W2kx for the comment on our review.

Reviewer_3ToCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Piecewise Bernstein Networks (PBN), a learnable activation function that represents nonlinear transformations using piecewise Bernstein polynomials. By ensuring continuity across intervals, PBN can exactly represent common activation functions such as ReLU and LeakyReLU while providing universal approximation capabilities for continuous functions. The proposed method is designed to be compatible with existing neural network architectures, including MLPs, CNNs, and Transformers, and employs optimized GPU kernels for efficient computation

Critical strengths

The paper presents a novel and interesting approach to designing learnable activation functions using piecewise Bernstein polynomials. One of its main strengths is that the proposed method is supported by mathematical proofs, which provide a solid theoretical foundation and strengthen the credibility of the approach. In particular, the theoretical analysis demonstrates the expressive capabilities of the proposed structure, including its ability to represent existing activation functions and approximate continuous functions. Another notable strength is its compatibility with a wide range of existing neural network architectures, such as MLPs, CNNs, and Transformers, without requiring substantial architectural modifications. This broad applicability enhances the practical relevance of the proposed method. Furthermore, the paper is well-organized and clearly written, with a logical presentation of the motivation, methodology, theoretical analysis, and experimental results, making it easy for readers to understand the proposed approach and its contributions.

Critical weaknesses

(1) The main concern with this paper is the mismatch between its theoretical findings and experimental evidence. The theoretical analysis emphasizes the importance of piecewise structures and establishes their universality and expressive capabilities. However, most experiments show that K=0, which is equivalent to the existing Bernstein activation, performs better than K>0. Although K>0 achieves better results in a few cases, these improvements are limited to specific settings. Therefore, the experimental results do not clearly support the practical significance of the proposed piecewise extension.

(2) Another concern is the lack of empirical evidence supporting the practical importance of kink representation. The paper highlights that the proposed PBN can exactly represent non-smooth activation functions, such as ReLU, by allowing discontinuities in the derivatives at knot locations. However, it remains unclear whether this capability actually contributes to the observed performance improvements. In particular, since many of the best-performing configurations use K=0, which does not introduce any internal knots, the experimental results do not demonstrate the necessity of kink representation. Additional ablation studies, such as comparing PBN with C⁰ and C¹ continuity constraints, would help clarify whether allowing kinks provides meaningful advantages in practice.

Questions for authors

(1) The proposed PBN uses fixed, uniformly spaced knots. However, the impact on model performance is not sufficiently explained. Can the authors clarify why uniformly spaced knots were chosen?

(2 Given that many of the best-performing configurations use K=0, can the authors clarify how the ability to represent kinks contributes to the performance improvements?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3ToC

We thank the reviewer for the careful reading and constructive feedback. The numbers referenced below are drawn from the updated table in our general response.

W1. Discrepancy between theoretical focus on piecewise structures and empirical results favouring K=0K=0. This was true of the submission, where KK was fixed before training. With dynamic KK (general response, item 2), the piecewise form becomes the most accurate PBN variant across all three parameter tiers in function fitting (0.34, 0.16, and 0.13×10−60.13\times10^{-6}, compared to 1.34, 0.40, and 0.21 for K=0K=0). It is also higher on the HIGGS MLP at all three tiers (.8225, .8552, and .8724, compared to .8206, .8545, and .8699), with a margin beyond the seed spread at 10510^{5} parameters. It achieves a higher FT-Transformer AUC (.8724 versus .8707), and its single ImageNet seed is 0.10 and 0.09 points above the three-seed median of K=0K=0 at the two network depths. In the other columns it is equal to the best submitted form or within its seed range. Overall, PBN with dynamic KK is the most accurate PBN form in 5 of the 9 columns of the table in our general response and equal in a sixth (and the most accurate in the 4 columns that are not in the table, namely function fitting and the HIGGS MLP at 10310^{3} and 10410^{4} parameters, which Figure 9 of the revised paper shows).

W2 and Q2. Practical importance of kink representation and C0C^{0} vs C1C^{1} continuity. The practical advantage of knots is local control, as each coefficient acts on one piece, or on two adjacent pieces when it is shared at a knot. To demonstrate this, we trained a single PBN unit of degree 3 to fit one-dimensional functions on [−5,5][-5,5] by optimizing only its coefficients. The table below displays the test MSE (median of three seeds) for varying KK values.

TargetK=0K=0K=1K=1K=3K=3K=7K=7
e−x/4cos⁡xe^{-x/4}\cos x0.564.4e-32.1e-46.0e-7
J0(4x)J_{0}(4x)0.0800.0560.0125.5e-4
e−x2sin⁡5xe^{-x^{2}}\sin 5x0.0590.0590.0552.8e-3
e−x2/8e^{-x^{2}/8}0.0128.7e-52.5e-72.4e-9

The error drops by factors ranging from 20 to over 10610^{6} as KK increases, consistent with the O(hn+1)O(h^{n+1}) rate of piecewise polynomials noted after Corollary 3.2.

We keep C0C^{0} joins for two reasons. First, C0C^{0} joins include all smoother joins as special cases. With dynamic KK, every newly inserted knot begins as a smooth join. Second, a C0C^{0} unit of degree 1 with a knot at the origin is exactly ReLU, LeakyReLU, or PReLU on its interval (Theorem 3.1). Any network that uses one of these activations can therefore be written as a PBN network with linear extension, with the same layers and the same weights. A C1C^{1} unit cannot represent these activations.

Q1. Rationale for uniformly spaced knots. Uniformly spaced knots allow the unit's evaluation to remain a fixed index computation within our fused GPU kernel. Additionally, with an odd KK, a knot aligns exactly at the origin, which is a strict requirement for exactly representing ReLU. We also tried knots learned by gradient together with the coefficients. The learned knots moved out of the region where the unit's inputs lie, and an epoch took 2.8 times as long, because variable knots cannot use the fused kernel (our Triton implementation of PBN). For these reasons, we stopped those runs and kept uniform knots. Dynamic KK preserves this uniformity during every knot insertion.

We thank the reviewer for recognizing that our approach is novel, theoretically grounded and compatible with existing architectures.

Below, we summarize the reviewer's suggestions and our response, which has further strengthened our work. We believe the additional analysis addresses all of the reviewer's questions.

  • Piecewise form against K=0K=0 (W1). With PBN with dynamic KK, the piecewise form is now the most accurate PBN form in most benchmark columns.
  • Kinks and C0C^{0} joins (W2 and Q2). A single PBN unit shows the error dropping sharply as KK increases, and C0C^{0} joins keep ReLU exactly representable.
  • Uniform knots (Q1). They keep the fused GPU kernel and a knot at the origin. Learned knots left the input region and were slower.

We appreciate the reviewer's feedback, which has helped us clarify the paper's contributions and strengthen its experimental support.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

General response to all reviewers

We thank the reviewers for the constructive feedback. We have made two additions since our initial submission, and our individual replies refer to the updated results in the table below.

1. A third seed on ImageNet-1k. We added a third seed for all ImageNet-1k experiments in response to Reviewers W2kx and 34JA, who asked for more seeds on ImageNet. Every ImageNet configuration is now reported consistently with our other settings, displaying the median over three seeds alongside the seed spread. Under this three-seed evaluation, PBN (K=0K=0) outperforms ReLU by 1.00 points on ResNet-34 and 0.61 points on ResNet-50 (where ReLU achieves 73.96±0.0773.96\pm0.07 and 77.05±0.0777.05\pm0.07, respectively). For each PBN form at its selected configuration, every seed exceeds every ReLU seed. The one PBN configuration that does not is K=0K=0 with n=9n=9 on ResNet-50, which stays at ReLU's level (77.02 against 77.05), as the paper reports.

2. PBN with dynamic KK (K→K‾K\rightarrow\overline{K}). We added this form in response to the reviewers. Reviewers 3ToC and 34JA asked for evidence of the practical benefit of the piecewise form, and Reviewer W2kx noted that KK still required a search. In our original submission, the number of interior knots KK was fixed prior to training. The dynamic KK variant uses the exact same PBN unit but accepts an upper bound K‾\overline{K} instead. During training, KK increases step by step up to K‾\overline{K}. At each step, a new knot is inserted at the midpoint of every piece. The coefficients of the new pieces are computed from the previous ones with de Casteljau's algorithm. The Bernstein form makes this exact and simple. Each unit computes exactly the same function immediately before and after the insertion. The network therefore computes the same function as well. The unit, its evaluation and the theory of Section 3 are unchanged.

Updated Results Each cell represents the validation-selected configuration, showing the median over three seeds ±\pm half the seed range (with three seeds, the standard deviation is 1.0 to 1.15 times this value). A dagger (†) indicates a single seed. For the two MLP settings, the table reports the 10510^{5}-parameter tier.

FFHIGGSR18 C10R18 C100R34R50FT-TViT C10ViT C100
PBN, K=0K=00.209±0.0140.209\pm0.014.8699±.0009.8699\pm.000994.86±0.2694.86\pm0.2675.09±0.3275.09\pm0.3274.97±0.0674.97\pm0.0677.66±0.1477.66\pm0.14.8707±.0005.8707\pm.000593.86±0.1093.86\pm0.1072.17±1.4872.17\pm1.48
PBN, fixed K≥1K\ge10.686±0.0340.686\pm0.034.8697±.0015.8697\pm.001594.72±0.0294.72\pm0.0275.61±0.1375.61\pm0.1374.89±0.0874.89\pm0.0877.58±0.0277.58\pm0.02.8717±.0006.8717\pm.000693.37±0.6593.37\pm0.6570.60±0.8370.60\pm0.83
PBN, dynamic KK0.135±0.0050.135\pm0.005.8724±.0007.8724\pm.000794.54±0.1094.54\pm0.1075.48±0.1675.48\pm0.1675.07†77.76†.8724±.0001.8724\pm.000193.87±0.1993.87\pm0.1971.63±0.9671.63\pm0.96
Other learnable0.125±0.0140.125\pm0.014.8669±.0009.8669\pm.000993.79±0.5393.79\pm0.5373.49±0.2073.49\pm0.2074.05±0.0474.05\pm0.0476.89±0.0276.89\pm0.02.8704±.0006.8704\pm.000692.19±0.7492.19\pm0.7470.47±0.3870.47\pm0.38
Fixed, ReLU-like15.1±0.715.1\pm0.7.8634±.0006.8634\pm.000694.75±0.1494.75\pm0.1475.66±0.2975.66\pm0.2974.19±0.2474.19\pm0.2477.31±0.1177.31\pm0.11.8720±.0008.8720\pm.000892.35±0.6192.35\pm0.6170.16±0.4170.16\pm0.41
Fixed, saturating7878±5207878\pm520.8568±.0005.8568\pm.0005not runnot run70.64±0.0670.64\pm0.0674.61±0.1074.61\pm0.10not run83.72±18.6683.72\pm18.6660.28±0.8560.28\pm0.85

FF is function fitting, test MSE ×10−6\times10^{-6} (geometric mean over 52 target functions, lower is better). HIGGS is the HIGGS MLP, test AUC. R18 is ResNet-18 on CIFAR, test top-1. R34 and R50 are ImageNet, validation top-1. FT-T is the FT-Transformer on HIGGS, test AUC. ViT is ViT-T/4 on CIFAR, test top-1. Other learnable covers KAN, Chebyshev, Fourier and Pi-Net (FF, HIGGS), and BerLU and GR-KAN (elsewhere). ReLU-like covers ReLU, GELU and ReGLU. Saturating is tanh.

When compared to the better of our two originally submitted forms in each column, PBN with dynamic KK reduces the function-fitting error from 0.21 to 0.13, raises the HIGGS MLP AUC from .8699 to .8724, and raises the FT-Transformer AUC from .8717 to .8724. On ImageNet, its single seed reaches 75.07 and 77.76, against medians of 74.97 and 77.66 for K=0K=0 over three seeds. It is equal on ViT CIFAR-10, and lower, within the seed range, on ViT CIFAR-100 and ResNet-18.

Similar trends appear at the 10310^{3} and 10410^{4} parameter tiers for both MLP settings. For function fitting, dynamic KK achieves errors of 0.34 and 0.16 versus 1.34 and 0.40 for K=0K=0. For HIGGS, it achieves .8225 and .8552 versus .8206 and .8545. As noted in the original submission, other learnable bases maintain higher accuracy than PBN for function fitting at these specific tiers (0.22 vs 0.34, and 0.10 vs 0.16). Figure 9 of the revised paper shows every activation separately in all 13 benchmark columns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.