acceptodds
Under review as a conference paper at ICLR 2027

Lightweight Explainable Physics-Informed Neural Networks by Learnable Activation Function

Abstract

Physics-informed neural networks (PINNs) provide a data-efficient framework for solving partial differential equations, but improving their accuracy often requires larger multilayer perceptron (MLP) backbones, increasing parameter count and computational cost. We investigate whether learnable activation functions can improve PINN performance while preserving lightweight backbones and explainable nonlinearities. We adapt the Cannistraci-Muscoloni-Gu generalized logistic-logit function (CMG) to PINNs and introduce CMG-GELU, which replaces the tanh term in the GELU approximation with learnable CMG. To support stable PINN training, we reformulate CMG's implicit logit phase into an explicit differentiable approximation and adopt a layer-wise parameterization that adds only two trainable activation parameters per hidden layer. We compare CMG and CMG-GELU with nine representative activation and architecture baselines across 25 forward and inverse PINN tasks using five random seeds. Compared with representative PINN method studies, our 25-task suite covers substantially more tasks, enabling a broader cross-task evaluation. CMG-GELU delivers the strongest aggregate performance and the highest cross-task consistency among the 11 methods, achieving the best mean rank, rank standard deviation, and Robustness Performance Ranking Index (RPRI), while CMG ranks second by RPRI. Learned activation parameters reveal task- and layer-dependent activation modulation patterns, providing parameter-level explainability, while low-dimensional parameter-profile embeddings provide exploratory task-level evidence of local organization among tasks with related physical regimes. These results support CMG-based learnable activations as a lightweight and explainable approach to improving PINNs across heterogeneous problems.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics-informed neural networks, learnable activation functions, adaptive activation functions, explainable neural networks, lightweight neural network modelling, partial differential equations, parameter-efficient neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o8LtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors suggest an architectural improvement to PINNs: They replace static general-purpose activation functions with an adaptive activation function, which is a form of GeLU where one can control both the strength of the step and its asymmetry. They used this approach on 25 PDE tasks, from 1d to 3d, covering both linear and non-linear equations. They compare their results to experiments they ran against 11 other approaches and show their average ranking- ranking top among these approaches

Critical strengths

The paper tackles an important problem in the field of PDE solvers -- how to improve PINN technology.

Critical weaknesses

It's hard to tell this paper apart from the surrounding noise, due to the following reasons:

  1. While within the ranking system they propose, they do end up top; the actual gains are within 20-30% of existing approaches, and occasionally, existing approaches also win.
  2. I believe the authors ran the other approaches themselves, but that also raises some concerns about hyperparameter choices, which adds to the overall noise concern. In particular, I've checked their results against published ones and found, for instance, that for Gray-Scott they report pirateNets reach 9.76e-02 whereas the original paper in Table 1. claims 3.61e-3, making it the winner for this case. For the Allen-Cahn equation, pirateNets report 2.24e-5, whereas the authors report that pirateNets reaches 3.23e-03. While I assume this was under different, legitimate settings (e.g. different optimizer/parameter-count), it again showcases that the significance of such order-one improvements factors, are hard to tell apart from the surrounding literature.
  3. There are other SOTA approaches out there, such as various KAN PINNs and SPINNs, which were not part of the comparison.

The general approach of allowing adjustable activation functions is not novel by itself (see KAN networks and KAN PINNs).

Questions for authors

Could you clarify the discrepancy between your reported results and those that appeared in the original work? Can you provide more details about the experimental pipeline?

3 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank the reviewer for this assessment.

To strengthen the reliability of our experimental evaluation, we adopt an improved pipeline that separates learning-rate selection from final testing. Specifically, learning rates are selected using an independent validation set and five initial seeds (0–4), followed by retraining from scratch with three fresh seeds (5–7) and evaluation on held-out test data. Furthermore, following the reviewer's suggestion to include additional SOTA approaches, we add PIKAN and SPINN (Shukla et al., 2024; Cho et al., 2023) as architecture baselines. We also introduce Adaptive GELU, a two-parameter learnable extension of GELU, in response to the other reviewers' suggestions. These additions expand our benchmark from 11 to 14 methods across 25 tasks.

We would first like to clarify that the primary advantage of CMG-GELU is not achieving the best performance on every individual task. Instead, its advantage is maintaining consistently strong performance across heterogeneous PINN problems. This is further supported by CMG-GELU achieving the best RPRI and mean rank, as well as the lowest rank SD among all 14 methods. These results establish a clear advantage in cross-task performance and consistency.

Moreover, rank-based metrics indicate which methods perform better, but do not capture the magnitude of error differences. We therefore report an additional metric named Normalized-Error Aggregate (NEA), where within each task, we divide each method's mean test error by the lowest mean test error among all 14 methods (1 indicates the best, and higher values indicate larger relative errors). For each method, we calculate the geometric mean of these normalized error ratios across 25 tasks.

The following table summarizes the aggregate results for CMG-GELU and the two newly introduced architecture baselines, with parenthetical values indicating rankings among all 14 methods.

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU1.87 (1)6.92 (1)4.56 (1)
PIKAN4.90 (13)13.48 (13)7.88 (11)
SPINN3.23 (12)12.24 (12)8.64 (13)

CMG-GELU achieves the lowest NEA (1.87), RPRI (6.92), and mean rank (4.56) among all 14 methods. The NEA results further indicate that its aggregate advantage extends beyond relative rankings to the magnitude of normalized errors across tasks. The updated manuscript details this expanded comparison in Table 4, with the improved evaluation protocol described in Section 2.3.

To further assess the reliability of these differences, we compute 95% bootstrap confidence intervals using 100,000 resamples. The following table reports pairwise differences in mean rank and RPRI, where negative values favor CMG-GELU.

ComparisonΔ Mean Rank (95% CI)Δ RPRI (95% CI)
CMG-GELU vs. PIKAN−3.32 [−5.72, −0.76]−6.56 [−8.47, −3.73]
CMG-GELU vs. SPINN−4.08 [−5.60, −2.36]−5.32 [−7.14, −3.76]

All reported confidence intervals exclude zero, providing additional statistical evidence that CMG-GELU achieves better aggregate ranking performance and cross-task consistency than both PIKAN and SPINN. Although other methods may outperform CMG-GELU on individual tasks, these results support its consistently competitive performance across heterogeneous PINN problems, rather than relying on a small number of task-specific improvements.

References

Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.

Cho, J., et al. (2023). Separable Physics-Informed Neural Networks. Advances in Neural Information Processing Systems, 36, 23761–23788.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 1

We thank the reviewer for this question. The discrepancy primarily reflects the differences in experimental settings. Our study focuses on lightweight PINNs, where all methods are evaluated under approximately matched parameter budgets, whereas the published PirateNet results (Wang et al., 2024) were obtained using substantially deeper and larger architectures, as well as different optimizers and training budgets. These differences are important because our objective is to assess the performance of activation functions and architectures under a shared lightweight setting, rather than reproduce the best results achievable by each method using its original large-scale configuration.

To clarify these differences, the following table compares our lightweight PirateNet implementation with the configurations reported in the PirateNet paper for the two tasks highlighted by the reviewer.

Allen–Cahn

ItemLightweight SettingOriginal Setting
Reported error3.23 e-032.24 e-05
Network size3 layers; width 259 layers; width 256
Parameter count2,202727,171
Activation functionTanhTanh
Optimizer and training budgetMuon: 40,000 steps, followed by 15,000 L-BFGS steps; 55,000 steps in totalAdam: 300,000 steps

Gray–Scott

ItemLightweight SettingOriginal Setting
Reported error9.76 e-023.61 e-03
Network size3 layers; width 499 layers; width 256
Parameter count8,039727,683
Activation functionTanhSwish
Optimizer and training budgetMuon: 20,000 steps, followed by 15,000 L-BFGS steps; 35,000 steps in totalAdam: 1,000,000 steps

As shown above, the published PirateNet configurations contain substantially more parameters and use different optimization protocols. The differences in reported errors (Allen–Cahn: 3.23e-3 versus 2.24e-5; Gray–Scott: 9.76e-2 versus 3.61e-3) should therefore be interpreted in the context of these distinct experimental settings, rather than as directly comparable results obtained under identical conditions. We have clarified this distinction in the revised manuscript to avoid any confusion between performance comparisons under lightweight parameter-matched settings and results reported under the original large-scale configurations.

Furthermore, to investigate whether CMG-GELU can also improve performance within the PirateNet architecture, we conducted additional experiments using the publicly released PirateNet code. We reproduced the Allen–Cahn and Gray–Scott experiments and replaced their original activation functions (tanh for Allen–Cahn and Swish for Gray–Scott) with CMG-GELU. The results are summarized below.

Allen–Cahn

Methodrelative L2 ↓
Reported2.24e-5
Reproduced1.02e-05
CMG-GELU8.32e-06

Gray–Scott

Methodrelative L2 ↓
Reported3.61e-3
Reproduced4.02e-4
CMG-GELU2.24e-4

CMG-GELU achieves lower relative L2 errors than the reproduced PirateNet baseline on both tasks, reducing error from 1.02e-5 to 8.32e-6 on Allen–Cahn and from 4.02e-4 to 2.24e-4 on Gray–Scott. These results provide additional evidence that CMG-GELU can improve performance beyond the lightweight MLP setting. The experimental configuration differences and results are documented in Appendix L of the updated manuscript.

References

Wang, S., Li, B., Chen, Y., & Perdikaris, P. (2024). PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. Journal of Machine Learning Research, 25(402), 1–51.

AuthorsOpenReview ↗

Response to Weakness 3

We thank the reviewer for this suggestion. In response, we have included PIKAN and SPINN as additional architecture baselines across all 25 tasks. As reported in our response to W1, CMG-GELU maintains the best NEA, RPRI, and mean rank among all 14 methods, including these two newly introduced baselines.

Regarding novelty, we would like to clarify that we do not claim to be the first to introduce learnable activation functions into PINNs. Rather, our contribution lies in the design of CMG-GELU, which incorporates geometrically interpretable, learnable nonlinear modulation into GELU while requiring only two additional trainable parameters per hidden layer, and in demonstrating its strong aggregate performance and cross-task consistency across heterogeneous PINN problems.

PIKAN achieves the highest top-1 count, but its performance shows high task selectivity. Its learnable univariate spline functions provide a flexible representation for smooth compositional structure, a central motivation behind KANs (Liu et al., 2025). This flexibility can deliver promising accuracy when the representation and optimization align with the problem. Previous differential-equation benchmarks have likewise documented substantial sensitivity of KAN-based models to problem settings (Shukla et al., 2024). Our results reveal a corresponding performance pattern: on the local low-order problems (including Heat, Diffusion–Reaction, Allen–Cahn, Fisher–KPP, Gray–Scott, Underdamped Vibration, Nonlinear Telegraph, and Klein–Gordon), PIKAN achieves a mean rank of 3.88, compared with 9.76 on the remaining tasks resulting in a significant difference (p = 0.0113). CMG-GELU’s corresponding mean ranks are 5.63 and 4.06 (p = 0.126) with no statistically significant difference detected between the two groups. On the remaining 17 tasks, CMG-GELU ranks ahead of PIKAN on 13 and achieves a significantly better mean rank (4.06 versus 9.76; paired p = 0.000763). PIKAN’s seven first-place finishes coexist with twelve bottom-three finishes, producing an overall mean rank of 7.88 and a Rank SD of 5.60. CMG-GELU combines three first-place finishes with no bottom-three finishes, achieving a better mean rank of 4.56 and a lower Rank SD of 2.36. Together, these results demonstrate CMG-GELU’s more consistently competitive performance across heterogeneous PINN problems.

References

Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.

Liu, Z., et al. (2025). KAN: Kolmogorov–Arnold Networks. International Conference on Learning Representations.

Reviewer_gx2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts the Cannistraci-Muscoloni-Gu generalized logistic-logit function (CMG) as a learnable activation for PINNs. It makes the logit phase explicit with a one-step Newton approximation, and uses one (μ,I)(\mu, I) pair per hidden layer (two extra parameters per layer). It also proposes CMG-GELU, which swaps the tanh in the GELU approximation for CMG. Eleven methods are compared on 25 forward and inverse PDE tasks, with five seeds and approximately matched parameter counts. CMG-GELU has the best mean rank (3.72) and the best author-defined RPRI (rank mean + rank SD). The authors also argue that the learned (μ,I)(\mu, I) values are “explainable” and that PCA embeddings of them cluster by physical family.

Critical strengths
  1. The paper compares 11 methods on 25 forward and inverse PDE tasks with five seeds, approximately matched parameter counts, a shared optimizer protocol, full per-task tables (E1-E9), and released code. Prior PINN activation and architecture papers report 2 to 10 tasks (Table 2), and PINN results are notoriously task-dependent, so this breadth gives the community something it can audit and reuse.

  2. CMG-GELU adds only two scalars per hidden layer, never ranks below 7th across the 25 tasks, and improves on tanh and fixed GELU on average. The authors explicitly claim aggregate robustness rather than per-task superiority and acknowledge the compute overhead and three-layer scope.

Critical weaknesses
  1. Activation-parameter methods (CMG, CMG-GELU, REAct, L-LAAF) search a 3×33 \times 3 learning-rate grid (9 configurations), while GELU, Sine, Swish, Sigmexd, the vanilla MLP and the architecture baselines search only 3. The winner is chosen by the lowest five-seed mean error against the reference solution, which is also the number reported. Best-of-9 versus best-of-3 on noisy 5-seed means gives a systematic optimistic bias, and it falls on the key comparison (CMG-GELU vs. GELU). Reference-based selection is also impossible in real forward-PINN use.
  2. There is also no ablation (fixed μ\mu and I,μI, \mu-only, II-only, or GELU with a learnable scale), so the gain cannot be attributed to CMG rather than to any two-parameters-per-layer adaptivity.
Questions for authors
  1. Appendix C gives CMG, CMG-GELU, REAct and L-LAAF a 3×33 \times 3 learning-rate grid (9 configurations) and the other methods 3. Selection uses the lowest five-seed mean error against the reference solution, which is also the reported number. Can you re-run with (a) equal numbers of configurations for all methods and (b) selection on a held-out criterion (a validation set or PDE residual), with final results on fresh seeds? Does the ranking, in particular CMG-GELU vs. GELU, survive? If it does, my main concern about the headline result largely goes away.

  2. What do the following give under the same protocol: CMG-GELU with fixed (μ,I)(\mu, I); μ\mu-only and II-only variants; and GELU with a two-parameter-per-layer learnable modification (for example, a learnable scale and shift inside the tanh)? Section 3.5 says II stays near 0.5 for CMG. Does II contribute anything measurable?

  3. Can you report bootstrap confidence intervals for mean rank and RPRI, a Friedman/Nemenyi (or similar) test, and a normalized-error aggregate or performance profile? Several tasks (Poisson L-shape, Darcy, Shallow Water, Fractional Poisson) look like floors or ties, so how do the ranks and RPRI change if these are excluded?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1 (Part 1/2)

We thank the reviewer for this question. We adopt an improved experimental pipeline that separates learning-rate selection from final evaluation using fresh random seeds and an independent test set. The protocol and results below examine whether our findings remain consistent under this improved evaluation procedure.

Training and Validation. We retain the training configurations described in manuscript, and for each task-method pair, we select the learning-rate configuration that minimizes the mean validation error across five initial seeds (0–4). We then freeze the selected learning rates and retrain each model from scratch using three fresh seeds (5, 6, and 7).

Evaluation on Held-out Test Set. For each task, we generate a new test set that is independent from the validation set. Reference values are obtained from analytical or numerical solutions, as applicable. The final results are computed exclusively from the three freshly trained models evaluated on these held-out test sets, ensuring that learning-rate selection and final performance reporting use separate random seeds and evaluation data.

Tied Learning Rate Search Budget. We would first like to clarify that the original LR search was balanced: all methods searched the same three backbone LR candidates. Methods with learnable activation parameters additionally searched activation-specific learning rates because these parameters may require different learning rates from those used for the backbone. Nevertheless, to address the reviewer's concern, we conduct an additional experiment with an equal search budget of nine configurations per method.

To equalize the candidate learning rate budget, we retain the 3x3 LR grid for learnable activation functions (CMG, CMG-GELU, REAct, L-LAAF, Adaptive GELU, PIKAN), and expand the original three LR candidates of other methods to nine logarithmically spaced candidates over the same range. The corresponding logarithmic sequence is

ηk=5×10−4×2k/4,k=0,1,…,8.\eta_k=5\times 10^{-4}\times 2^{k/4},\qquad k=0,1,\ldots,8.

which divides each original interval into four equal logarithmic steps and retains the original candidates at k=0,4,8. For Inverse Poisson, the sequence is scaled down by a factor of ten, consistent with its original LR search range. Consequently, all methods are evaluated with the same number of LR candidate configurations. We refer to the original LR search strategy as "Original" and the equal-budget strategy as "Tied-budget". Both settings follow the improved experimental pipeline described above.

AuthorsOpenReview ↗

Response to Weakness 1 and Question 1 (Part 2/2)

Additional Baselines. Following reviewer’s suggestion in Q2, we introduce Adaptive GELU, which incorporates learnable scale and shift parameters into the tanh term of GELU. It also follows the layer-wise setting as CMG-GELU, where each layer has 2 learnable activation function parameters (a and b in this case)

AdaptiveGELU⁡(x)=x2[1+tanh⁡(a2π(x+0.044715x3)+b)]\operatorname{AdaptiveGELU}(x)=\frac{x}{2}\left[1+\tanh\left(a\sqrt{\frac{2}{\pi}}\left(x+0.044715x^{3}\right)+b\right)\right]

Furthermore, following Reviewer o8Lt's suggestion, we include two additional SOTA baselines, PIKAN and SPINN (Shukla et al., 2024; Cho et al., 2023). This expands our benchmark from 11 to 14 methods across all 25 tasks.

Results. To complement rank-based metrics, we introduce the Normalized-Error Aggregate (NEA) as suggested by the reviewer, which also accounts for the magnitude of performance differences. For each task, we normalize each method's mean test error by the lowest mean test error among all 14 methods on that task. The NEA is then computed as the geometric mean of these normalized errors across the 25 tasks. A value of 1 indicates task-wise optimal performance throughout the benchmark, and lower values indicate better aggregate performance.

The following tables summarize the results under both LR search settings. Parenthetical values indicate rankings among all 14 methods for each metric. Lower values indicate better performance for NEA, RPRI, and mean rank.

Original LR Search Setting

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU1.87 (1)6.92 (1)4.56 (1)
GELU2.27 (2)9.78 (2)6.56 (3)
Adaptive GELU2.53 (6)10.06 (4)6.84 (4)

Tied-Budget LR Search Setting

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU1.95 (1)7.28 (1)4.80 (1)
GELU2.56 (3)9.63 (2)6.56 (3)
Adaptive GELU2.64 (6)9.86 (4)6.84 (4)

CMG-GELU maintains the best NEA, RPRI, and mean rank among all 14 methods under both settings. In particular, it consistently outperforms GELU and Adaptive GELU, even when all methods receive the same number of LR candidate configurations. These results support the robustness of our main finding and indicate that CMG-GELU's advantage cannot be attributed solely to the larger LR search budget.

The updated manuscript documents the revised evaluation protocol in Section 2.3, the main benchmark results in Table 4, and the tied-budget analysis in Appendix J.

References

Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.

Cho, J., et al. (2023). Separable Physics-Informed Neural Networks. Advances in Neural Information Processing Systems, 36, 23761–23788.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

We thank the reviewer for highlighting this point. We conduct five ablation experiments for both CMG and CMG-GELU across all 25 tasks under the Original LR-search setting, using the improved experimental pipeline described above. The ablation configurations are as follows:

  • Fixed- mu/I: we freeze the learned mu and I values, and train the model from scratch. This examines whether fixing the learned activation shapes can reproduce the performance of jointly optimizing the activation parameters and network weights.

  • Mu-only: we freeze I at 0.5, and train mu. This isolates the contribution of learning I

  • I-only (mu=0.25): we freeze mu at 0.25 (logistic-phase), and train I. This is to investigate the significance of training mu

  • I-only (mu=0.75): we freeze mu at 0.75 (logit-phase), and train I. This is to investigate the significance of training mu

  • Logistic-only: We restrict CMG to the logistic phase to assess whether access to the logit phase contributes to performance.

In addition, the Adaptive GELU baseline, introduced in our response to W1/Q1, directly addresses whether a two-parameter-per-layer modification of GELU can achieve comparable performance, and its results can be found in our response above.

For each ablation, we replace the corresponding original method with its ablated variant while keeping the other 13 methods unchanged. We then recompute the aggregate metrics across all 25 tasks. The following table summarizes the NEA, RPRI and Mean Rank results. Parenthetical values indicate ranks among the 14 methods within each corresponding comparison.

CMG

ConfigurationNEA ↓RPRI ↓Mean Rank ↓
Original2.47 (3)10.53 (6)6.84 (4)
μ-only: fixed I=0.52.78 (9)10.82 (7)7.48 (8)
I-only: fixed μ=0.253.06 (11)11.38 (8)8.00 (11)
I-only: fixed μ=0.753.24 (12)11.97 (10)8.88 (13)
Fixed learned μ/I2.82 (10)11.00 (7)8.00 (11)
Logistic-only2.92 (11)11.07 (7)7.40 (7)

CMG-GELU

ConfigurationNEA ↓RPRI ↓Mean Rank ↓
Original1.87 (1)6.92 (1)4.56 (1)
μ-only: fixed I=0.52.27 (2)8.61 (1)5.96 (1)
I-only: fixed μ=0.252.22 (1)8.43 (1)5.48 (1)
I-only: fixed μ=0.752.60 (2)10.76 (7)6.80 (5)
Fixed learned μ/I2.08 (1)8.50 (1)5.28 (1)
Logistic-only1.87 (1)7.47 (1)4.68 (1)

The full CMG-GELU model achieves the lowest RPRI among all its ablation configurations. Its NEA is also lower than those of the ablated variants, except for logistic-only, which matches the full model at the reported precision (1.87), but has a higher RPRI (7.47 vs. 6.92). Similar trends are observed for CMG, whose full model achieves the lowest NEA and RPRI among its variants.

Regarding the contribution of I, fixing I at 0.5 while learning μ increases NEA from 2.47 to 2.78 for CMG and from 1.87 to 2.27 for CMG-GELU. This indicates that learning I provides measurable benefits. Furthermore, fixing μ at either 0.25 or 0.75 also worsens aggregate performance, supporting the benefit of jointly learning both shape parameters.

Finally, as reported in our response to W1/Q1, CMG-GELU also outperforms Adaptive GELU under both Original and Tied-budget settings, despite both methods introducing only two learnable activation parameters per hidden layer. Together, these results suggest that CMG-GELU's improvements cannot be explained solely by the addition of learnable parameters, but are associated with its adaptive nonlinear shape modulation.

The detailed ablation configurations and results are provided in Appendix K of the updated manuscript.

AuthorsOpenReview ↗

Response to Question 3

We thank the reviewer for this suggestion. We conduct additional statistical and robustness analyses under both the Original and Tied-budget LR-search settings, as detailed below.

Bootstrap CI

We compute 95% confidence intervals using 100,000 bootstrap resamples for the differences in mean rank and RPRI between CMG-GELU and two representative baselines, GELU and Adaptive GELU. Negative differences mean CMG-GELU has a better mean rank and RPRI than the corresponding baseline.

Original LR Search Setting

ComparisonΔ Mean Rank (95% CI)Δ RPRI (95% CI)
CMG-GELU vs. GELU−2.00 [−3.52, −0.48]−2.85 [−4.46, −1.28]
CMG-GELU vs. Adaptive GELU−2.28 [−4.12, −0.60]−3.14 [−5.47, −0.96]

Tied-budget LR Search Setting

ComparisonΔ Mean Rank (95% CI)Δ RPRI (95% CI)
CMG-GELU vs. GELU−1.76 [−3.20, −0.24]−2.35 [−3.78, −0.56]
CMG-GELU vs. Adaptive GELU−2.04 [−3.88, −0.28]−2.58 [−4.97, −0.38]

All reported confidence intervals exclude zero, consistently favoring CMG-GELU over GELU and Adaptive GELU in both mean rank and RPRI under both LR-search settings.

Friedman test

We perform a Friedman test (Demšar, 2006) on the task-wise rankings of all 14 methods across 25 tasks. The results indicate statistically significant differences among methods under both the Original (p = 1.74 × 10⁻⁵) and Tied-budget (p = 2.14 × 10⁻⁵) settings.

Normalized Error Aggregate

As defined in our response to W1/Q1, NEA normalizes each method's error by the lowest error within each task and aggregates these ratios using the geometric mean across tasks. Unlike rank-based metrics, NEA also accounts for the magnitude of error differences. CMG-GELU achieves the lowest NEA among all 14 methods under both the Original (1.87) and Tied-budget (1.95) settings, further supporting its strong aggregate performance.

Excluding the four specified tasks

Following the reviewer's suggestion, we exclude Poisson L-shape, Darcy Flow, Shallow Water, and Fractional Poisson, and recompute the aggregate metrics over the remaining 21 tasks. The results are summarized below. Parenthetical values indicate ranks among all 14 methods.

Original LR Search Setting — 21 Tasks

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU2.09 (1)7.13 (1)4.76 (1)
GELU2.59 (2)10.00 (4)6.76 (4)
Adaptive GELU2.98 (7)10.35 (6)7.19 (7)

Tied-budget LR Search Setting — 21 Tasks

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU2.19 (1)7.53 (1)5.00 (1)
GELU2.97 (2)9.77 (2)6.67 (4)
Adaptive GELU3.13 (7)10.20 (5)7.14 (6)

CMG-GELU retains the best NEA, RPRI, and mean rank among all 14 methods under both settings. These results demonstrate that its leading aggregate performance does not depend on the inclusion of the four specified tasks.

The 21-task results are also detailed in Tables I3 and J4 of the updated manuscript.

References

Demšar, J. (2006). Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7(1), 1–30.

Reviewer_X9MWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, authors propose learnable activation functions for PINNs based on CMG function. Authors make the logit phase of CMG explicit with a one-step Newton approximation, share the two shape parameters within each hidden layer, and introduce CMG-GELU, which replaces the tanh term in the GELU approximation with CMG. Methods are compared with activation and architecture baselines on a benchmark of 25 PDEs, and the learned parameters are analyzed as a source of explainability. The work closely repeats existing work on CMG, and the proposed method does not outperform its competitors on most tasks.

Critical strengths

I can highlight following strengths of the paper:

  1. Authors assemble a benchmark of 25 tasks. This is a comprehensive range for validating the methodology and a useful contribution that other researchers can reuse.
  2. The experimental setup is aligned and documented for every task and every baseline. The paper provides detailed reports of the results.
  3. Theoretical derivations presented in the paper are correct.
Critical weaknesses

Despite the merits of the paper, I must emphasize the following weaknesses:

  1. The CMG function is taken from [1]. Moreover, that work already proposes using the distribution of the learnable μ\mu and II as an explainability tool, including for PINNs. [2] already uses a one-step Newton scheme to resolve the automatic differentiation issue in CMG. It also shows that applying CMG layer-wise is better than neuron-wise, which is another conclusion of this submission. Moreover, CMG is already adapted for PINNs in [2], and [2] already studies the explainability of this activation. Thus, the novelty of the authors reduces to collecting PDEs from different papers into a benchmark and constructing the CMG-GELU activation.
  2. CMG-GELU substitutes CMG for tanh inside the GELU approximation. The paper does not explain why CMG should be useful in this particular place. Why did the authors not consider simply GELU with two learnable scalars, for example a slope and a shift in the style of L-LAAF?
  3. I also have concerns about the comparison of CMG-GELU with its competitors. By the introduced RPRI metric, the proposed method is the best, but its task-wise performance is rather weak. CMG-GELU is the leader on only 16% of tasks, and in 48% of tasks it does not even reach the top 3. If PirateNet and Fourier are excluded, these numbers become 28% and 40%, respectively. Thus, even among other activations, CMG-GELU has no clear advantage. Combined with the fact that CMG-GELU requires an order of magnitude higher peak GPU memory than other activations, the method seems useless in practice.

Moreover, Figure 1 in this submission is a copy of Figure 1 in [2] and Figure 1 in [1]. This raises concerns about the originality of this submission.


References:

[1] Gu, Wenqi, et al. "A generalized logistic-logit function and its application to multi-layer perceptron and neuron segmentation." Frontiers in Artificial Intelligence".

[2] Gu, Wenqi, and Carlo Vittorio Cannistraci. "Explainability of Node Modulation in Classical and Physics-Informed Neural Networks via a Generalized Activation Function."

Questions for authors
  1. Why was CMG embedded specifically into GELU? Why should GELU be combined with CMG rather than with another learnable activation?
7 comments
AuthorsOpenReview ↗

Response to Weakness 1

Dear Reviewer, regarding this point since it is an ethical concern we can't reply yet extensively, because we are waiting for a confirmation about the appropriate reply from Program Chair, Senior Area Chair and Area Chair that we promptly contacted to coordinate a reply compliant with ICLR policy.

As a consequence, your reviews, comments and scores should reflect the evaluation of our study per se, independently from any related pre-print, please.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 1

We thank the reviewer for this constructive question. Our motivation for embedding CMG into GELU is twofold. First, GELU is the strongest fixed-activation baseline in our benchmark, making it a natural starting point for further improvement. Second, CMG provides a more flexible sigmoid-shaped nonlinearity than the fixed tanh function. By replacing tanh with CMG, we retain GELU's smooth gated-linear structure while allowing its nonlinear gating behavior to adapt across tasks and hidden layers through two geometrically interpretable parameters, μ and I.

Following the reviewer's suggestion, we additionally evaluate Adaptive GELU, which introduces a learnable scale and shift inside tanh. Like CMG-GELU, it employs two learnable activation parameters (a and b) per hidden layer, providing a direct comparison with the same activation-parameter budget.

AdaptiveGELU⁡(x)=x2[1+tanh⁡(a2π(x+0.044715x3)+b)]\operatorname{AdaptiveGELU}(x)=\frac{x}{2}\left[1+\tanh\left(a\sqrt{\frac{2}{\pi}}\left(x+0.044715x^{3}\right)+b\right)\right]

In addition, following Reviewer o8Lt's suggestion to include 2 additional PINN baselines (Shukla et al., 2024; Cho et al., 2023), we incorporate PIKAN and SPINN, expanding our benchmark from 11 to 14 methods across all 25 tasks.

For these additional experiments, we adopt an improved evaluation pipeline: learning rates are selected using an independent validation set and five initial seeds (0–4), followed by retraining from scratch with three fresh seeds (5–7) and evaluation on test data.

Moreover, rank-based metrics indicate which methods perform better, but not by how much. To account for the magnitude of performance differences, we additionally report the Normalized-Error Aggregate (NEA). For each task, we normalize each method's mean test error by the lowest mean test error among all methods, so that a value of 1 indicates the best performance and higher values indicate larger errors relative to the task-wise best. We then compute the geometric mean of these normalized errors across all 25 tasks. Lower NEA indicates better aggregate performance. The following table summarizes NEA, RPRI, and mean rank, with parenthetical values indicating rankings among all 14 methods.

MethodNEA ↓RPRI ↓Mean Rank ↓
CMG-GELU1.87 (1)6.92 (1)4.56 (1)
GELU2.27 (2)9.78 (2)6.56 (3)
Adaptive GELU2.53 (6)10.06 (4)6.84 (4)

CMG-GELU achieves the lowest NEA, RPRI, and mean rank among all 14 methods, demonstrating both stronger aggregate accuracy and greater cross-task consistency. These results suggest that CMG-GELU's improvement cannot be explained merely by adding two learnable parameters per hidden layer, but is associated with the flexibility of CMG's nonlinear shape modulation. We have updated the manuscript to incorporate the expanded benchmark and improved evaluation protocol (Sections 2.1–2.3), with the updated results detailed in Table 4.

References

Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.

Cho, J., et al. (2023). Separable Physics-Informed Neural Networks. Advances in Neural Information Processing Systems, 36, 23761–23788.

AuthorsOpenReview ↗

Response to Weakness 3

We respectfully disagree with the reviewer’s concern about CMG-GELU's task-wise performance as weak.

As explicitly stated in the article, the advantage of CMG-GELU is not achieving best performance on every individual task, which is also not possible for other methods. Instead, its advantage is maintaining consistently strong performance across heterogeneous PINN problems. This is further supported by CMG-GELU achieving the best NEA, RPRI and mean rank among all 14 methods across 25 tasks. These results establish a clear advantage in cross-task performance and consistency.

Furthermore, using the top-3 count criterion proposed by the reviewer, we compare the top-3 task count between different methods. CMG-GELU ranks among the top three on 9 of 25 tasks, the second-highest count among all 14 methods, behind PIKAN (10) and ahead of PirateNet (8) and GELU (5). The following table summarizes the relevant performance metrics, with parenthetical values indicating ranks among all 14 methods.

MethodNEA ↓RPRI ↓Mean Rank ↓Top-1 ↑Top-3 ↑
CMG-GELU1.87 (1)6.92 (1)4.56 (1)3 (2)9 (2)
GELU2.27 (2)9.78 (2)6.56 (3)1 (8)5 (6)
PirateNet2.52 (5)9.79 (3)6.20 (2)2 (4)8 (3)
PIKAN4.90 (13)13.48 (13)7.88 (11)7 (1)10 (1)
Fourier33.58 (14)16.09 (14)11.72 (14)1 (8)3 (12)

Although PIKAN achieves the highest number of first-place finishes (7), it also ranks among the bottom three on 12 tasks, indicating substantial task-dependent variability. Our analysis reveals that PIKAN performs particularly well on eight local, low-order problems (Heat, Diffusion–Reaction, Allen–Cahn, Fisher–KPP, Gray–Scott, Underdamped Vibration, Nonlinear Telegraph, and Klein–Gordon), achieving a mean rank of 3.88, compared with 9.76 on the remaining 17 tasks. CMG-GELU shows a more balanced pattern, with corresponding mean ranks of 5.63 and 4.06, and outperforms PIKAN on 13 of the remaining 17 tasks. This suggests that PIKAN's flexible spline-based representation can be highly effective for certain problem classes, but its advantage does not generalize uniformly across heterogeneous tasks. In contrast, CMG-GELU maintains more consistent performance across the benchmark.

As introduced in our response to W2/Q1, rank-based metrics indicate relative ordering but do not capture the magnitude of error differences. We therefore additionally report the Normalized-Error Aggregate (NEA). CMG-GELU achieves the lowest NEA (1.87) among all 14 methods, further supporting its aggregate performance advantage beyond rank-based measures.

Regarding the claim that CMG-GELU requires "an order of magnitude higher peak GPU memory than other activations", we would like to clarify that the measurements in Appendix F, Table F2, do not support this reviewer’s comment. CMG-GELU requires 0.106 GiB of peak GPU memory, compared with 0.080 GiB for GELU and 0.124 GiB for REAct. Thus, its peak memory usage is approximately 1.33 times that of GELU, rather than an order of magnitude higher.

Furthermore, we have optimized the implementation to reduce computational overhead. Previously, the required input derivatives were computed using nested automatic differentiation. In the optimized implementation, we directly propagate these derivatives through the chain rule and use torch.compile to reduce computational and GPU kernel-launch overhead. We benchmarked both implementations on an NVIDIA RTX 5090 under the same experimental conditions on Burger’s Equation task, with the results summarized below.

MethodPrevious Time (min) ↓Optimized Time (min) ↓Previous GPU Memory (MB) ↓Optimized GPU Memory (MB) ↓
CMG-GELU13.92 ± 0.408.44 ± 0.16114.06106.49
GELU9.19 ± 0.047.79 ± 0.1285.9279.73
Vanilla MLP8.88 ± 0.047.61 ± 0.2080.9178.96
PirateNet12.45 ± 0.119.98 ± 0.2489.5583.51

For CMG-GELU, the measured training time decreases from 13.92 to 8.44 minutes, representing a 39.4% reduction, while peak GPU memory decreases from 114.06 to 106.49 MB, well within the memory capacity of most modern GPUs. After optimization, its training time has approached near-vanilla-MLP level (8.44 vs. 7.61 minutes). These results further demonstrate CMG-GELU’s practical applicability. Detailed results are provided in Appendix H of the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.