- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors suggest an architectural improvement to PINNs: They replace static general-purpose activation functions with an adaptive activation function, which is a form of GeLU where one can control both the strength of the step and its asymmetry. They used this approach on 25 PDE tasks, from 1d to 3d, covering both linear and non-linear equations. They compare their results to experiments they ran against 11 other approaches and show their average ranking- ranking top among these approaches
Critical strengths
The paper tackles an important problem in the field of PDE solvers -- how to improve PINN technology.
Critical weaknesses
It's hard to tell this paper apart from the surrounding noise, due to the following reasons:
- While within the ranking system they propose, they do end up top; the actual gains are within 20-30% of existing approaches, and occasionally, existing approaches also win.
- I believe the authors ran the other approaches themselves, but that also raises some concerns about hyperparameter choices, which adds to the overall noise concern. In particular, I've checked their results against published ones and found, for instance, that for Gray-Scott they report pirateNets reach 9.76e-02 whereas the original paper in Table 1. claims 3.61e-3, making it the winner for this case. For the Allen-Cahn equation, pirateNets report 2.24e-5, whereas the authors report that pirateNets reaches 3.23e-03. While I assume this was under different, legitimate settings (e.g. different optimizer/parameter-count), it again showcases that the significance of such order-one improvements factors, are hard to tell apart from the surrounding literature.
- There are other SOTA approaches out there, such as various KAN PINNs and SPINNs, which were not part of the comparison.
The general approach of allowing adjustable activation functions is not novel by itself (see KAN networks and KAN PINNs).
Questions for authors
Could you clarify the discrepancy between your reported results and those that appeared in the original work? Can you provide more details about the experimental pipeline?
3 comments
Response to Weakness 2 and Question 1
We thank the reviewer for this question. The discrepancy primarily reflects the differences in experimental settings. Our study focuses on lightweight PINNs, where all methods are evaluated under approximately matched parameter budgets, whereas the published PirateNet results (Wang et al., 2024) were obtained using substantially deeper and larger architectures, as well as different optimizers and training budgets. These differences are important because our objective is to assess the performance of activation functions and architectures under a shared lightweight setting, rather than reproduce the best results achievable by each method using its original large-scale configuration.
To clarify these differences, the following table compares our lightweight PirateNet implementation with the configurations reported in the PirateNet paper for the two tasks highlighted by the reviewer.
Allen–Cahn
| Item | Lightweight Setting | Original Setting |
|---|---|---|
| Reported error | 3.23 e-03 | 2.24 e-05 |
| Network size | 3 layers; width 25 | 9 layers; width 256 |
| Parameter count | 2,202 | 727,171 |
| Activation function | Tanh | Tanh |
| Optimizer and training budget | Muon: 40,000 steps, followed by 15,000 L-BFGS steps; 55,000 steps in total | Adam: 300,000 steps |
Gray–Scott
| Item | Lightweight Setting | Original Setting |
|---|---|---|
| Reported error | 9.76 e-02 | 3.61 e-03 |
| Network size | 3 layers; width 49 | 9 layers; width 256 |
| Parameter count | 8,039 | 727,683 |
| Activation function | Tanh | Swish |
| Optimizer and training budget | Muon: 20,000 steps, followed by 15,000 L-BFGS steps; 35,000 steps in total | Adam: 1,000,000 steps |
As shown above, the published PirateNet configurations contain substantially more parameters and use different optimization protocols. The differences in reported errors (Allen–Cahn: 3.23e-3 versus 2.24e-5; Gray–Scott: 9.76e-2 versus 3.61e-3) should therefore be interpreted in the context of these distinct experimental settings, rather than as directly comparable results obtained under identical conditions. We have clarified this distinction in the revised manuscript to avoid any confusion between performance comparisons under lightweight parameter-matched settings and results reported under the original large-scale configurations.
Furthermore, to investigate whether CMG-GELU can also improve performance within the PirateNet architecture, we conducted additional experiments using the publicly released PirateNet code. We reproduced the Allen–Cahn and Gray–Scott experiments and replaced their original activation functions (tanh for Allen–Cahn and Swish for Gray–Scott) with CMG-GELU. The results are summarized below.
Allen–Cahn
| Method | relative L2 ↓ |
|---|---|
| Reported | 2.24e-5 |
| Reproduced | 1.02e-05 |
| CMG-GELU | 8.32e-06 |
Gray–Scott
| Method | relative L2 ↓ |
|---|---|
| Reported | 3.61e-3 |
| Reproduced | 4.02e-4 |
| CMG-GELU | 2.24e-4 |
CMG-GELU achieves lower relative L2 errors than the reproduced PirateNet baseline on both tasks, reducing error from 1.02e-5 to 8.32e-6 on Allen–Cahn and from 4.02e-4 to 2.24e-4 on Gray–Scott. These results provide additional evidence that CMG-GELU can improve performance beyond the lightweight MLP setting. The experimental configuration differences and results are documented in Appendix L of the updated manuscript.
References
Wang, S., Li, B., Chen, Y., & Perdikaris, P. (2024). PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. Journal of Machine Learning Research, 25(402), 1–51.
Response to Weakness 3
We thank the reviewer for this suggestion. In response, we have included PIKAN and SPINN as additional architecture baselines across all 25 tasks. As reported in our response to W1, CMG-GELU maintains the best NEA, RPRI, and mean rank among all 14 methods, including these two newly introduced baselines.
Regarding novelty, we would like to clarify that we do not claim to be the first to introduce learnable activation functions into PINNs. Rather, our contribution lies in the design of CMG-GELU, which incorporates geometrically interpretable, learnable nonlinear modulation into GELU while requiring only two additional trainable parameters per hidden layer, and in demonstrating its strong aggregate performance and cross-task consistency across heterogeneous PINN problems.
PIKAN achieves the highest top-1 count, but its performance shows high task selectivity. Its learnable univariate spline functions provide a flexible representation for smooth compositional structure, a central motivation behind KANs (Liu et al., 2025). This flexibility can deliver promising accuracy when the representation and optimization align with the problem. Previous differential-equation benchmarks have likewise documented substantial sensitivity of KAN-based models to problem settings (Shukla et al., 2024). Our results reveal a corresponding performance pattern: on the local low-order problems (including Heat, Diffusion–Reaction, Allen–Cahn, Fisher–KPP, Gray–Scott, Underdamped Vibration, Nonlinear Telegraph, and Klein–Gordon), PIKAN achieves a mean rank of 3.88, compared with 9.76 on the remaining tasks resulting in a significant difference (p = 0.0113). CMG-GELU’s corresponding mean ranks are 5.63 and 4.06 (p = 0.126) with no statistically significant difference detected between the two groups. On the remaining 17 tasks, CMG-GELU ranks ahead of PIKAN on 13 and achieves a significantly better mean rank (4.06 versus 9.76; paired p = 0.000763). PIKAN’s seven first-place finishes coexist with twelve bottom-three finishes, producing an overall mean rank of 7.88 and a Rank SD of 5.60. CMG-GELU combines three first-place finishes with no bottom-three finishes, achieving a better mean rank of 4.56 and a lower Rank SD of 2.36. Together, these results demonstrate CMG-GELU’s more consistently competitive performance across heterogeneous PINN problems.
References
Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.
Liu, Z., et al. (2025). KAN: Kolmogorov–Arnold Networks. International Conference on Learning Representations.
Response to Weakness 1
We thank the reviewer for this assessment.
To strengthen the reliability of our experimental evaluation, we adopt an improved pipeline that separates learning-rate selection from final testing. Specifically, learning rates are selected using an independent validation set and five initial seeds (0–4), followed by retraining from scratch with three fresh seeds (5–7) and evaluation on held-out test data. Furthermore, following the reviewer's suggestion to include additional SOTA approaches, we add PIKAN and SPINN (Shukla et al., 2024; Cho et al., 2023) as architecture baselines. We also introduce Adaptive GELU, a two-parameter learnable extension of GELU, in response to the other reviewers' suggestions. These additions expand our benchmark from 11 to 14 methods across 25 tasks.
We would first like to clarify that the primary advantage of CMG-GELU is not achieving the best performance on every individual task. Instead, its advantage is maintaining consistently strong performance across heterogeneous PINN problems. This is further supported by CMG-GELU achieving the best RPRI and mean rank, as well as the lowest rank SD among all 14 methods. These results establish a clear advantage in cross-task performance and consistency.
Moreover, rank-based metrics indicate which methods perform better, but do not capture the magnitude of error differences. We therefore report an additional metric named Normalized-Error Aggregate (NEA), where within each task, we divide each method's mean test error by the lowest mean test error among all 14 methods (1 indicates the best, and higher values indicate larger relative errors). For each method, we calculate the geometric mean of these normalized error ratios across 25 tasks.
The following table summarizes the aggregate results for CMG-GELU and the two newly introduced architecture baselines, with parenthetical values indicating rankings among all 14 methods.
CMG-GELU achieves the lowest NEA (1.87), RPRI (6.92), and mean rank (4.56) among all 14 methods. The NEA results further indicate that its aggregate advantage extends beyond relative rankings to the magnitude of normalized errors across tasks. The updated manuscript details this expanded comparison in Table 4, with the improved evaluation protocol described in Section 2.3.
To further assess the reliability of these differences, we compute 95% bootstrap confidence intervals using 100,000 resamples. The following table reports pairwise differences in mean rank and RPRI, where negative values favor CMG-GELU.
All reported confidence intervals exclude zero, providing additional statistical evidence that CMG-GELU achieves better aggregate ranking performance and cross-task consistency than both PIKAN and SPINN. Although other methods may outperform CMG-GELU on individual tasks, these results support its consistently competitive performance across heterogeneous PINN problems, rather than relying on a small number of task-specific improvements.
References
Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. Computer Methods in Applied Mechanics and Engineering, 431, 117290.
Cho, J., et al. (2023). Separable Physics-Informed Neural Networks. Advances in Neural Information Processing Systems, 36, 23761–23788.