- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an improved PINNs architecture, beignet, which targets the difficulties of standard PINNs: struggle to achieve highly accurate solutions and their computation scales poorly with parameter count. Beignet method is based on the random Fourier feature (RFF) input embedding method in PINNs, which replaces it with a learned multi-resolution feature pyramid. It uses Fourier interpolation at each level of the pyramid to embed the input coordinates and send the features into the network. By directly controlling the representation bandlimit, the method addresses the drawback of RFF. Experiments show that it consistently outperforms several SOTA methods on multiple challenging benchmarks.
Critical strengths
- This paper addresses the drawback of RFF in PINNs, where RFF is a standard tool for PINNs for mitigating the spectral bias problems.
- This paper has a clear presentation and motivation, and the numerical results show the effectiveness of the method on multiple benchmarks.
- This paper provides a method that can be easily combined with existing methods and is effective for a wide range of problems.
Critical weaknesses
- This paper lacks theoretical analysis and spectral-bias benchmarks. The abstract/intro/background of the paper emphasizes spectral bias and the bandlimit drawback of RFF, and introduces the NTK in the background section. However, the rest of the paper does not analyze the NTK of the proposed method theoretically. It also provides no other theoretical results supporting the method's effectiveness, and few popular spectral-bias / high-frequency benchmarks.
- The main benchmarks are mostly periodic boundary problems, such as Allen–Cahn and KdV, which suit the structure of the method particularly well from a theoretical view. Even experiments on other benchmarks are included in later parts, but the baselines chosen for comparison are not SOTA for those benchmarks.
- The paper only states that it follows the JAX-PI training procedure, without specifying the training hyperparameters, data, and other detailed settings. JAX-PI is already an advanced structure combined with advanced training techniques.
Questions for authors
- For RFF-PINNs, the bandwidth is usually chosen according to the frequency domain of the given PDE, which has some theoretical guarantees based on the NTK theory. Could the authors explain how the hyperparameters of beignet are chosen, e.g., the number of levels and the parameter initialization?
- The paper states that they use techniques such as ModifiedMLP and causal time-chunk weighting based on JAX-PI training. Can the author ensure that the advantage of beignet persists when these components are removed? Can beignet stand as a relatively independent algorithm, or must it be used together with these methods?
- The choice of benchmarks and baselines could be better aligned with spectral bias and improvement on RFF methods if the author wants to emphasize it. For the benchmarks, could the authors try some popular high-frequency problems such as high-frequency heat equations or maybe high-frequency Helmholtz equations? For the baselines, the RFF methods should be compared more, rather than mostly in appendix, such as [1] and [2]. 4.In the Temporal Pyramid Blending section, is piecewise linear in time. Would this property make the method unsuitable for equations involving second- or higher-order time derivatives? E.g., in the wave equation.
[1] Wang, Sifan, Hanwen Wang, and Paris Perdikaris. "On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks." Computer Methods in Applied Mechanics and Engineering 384 (2021): 113938.
[2] Wu, Yulun, Aguiar, M., Karl H. Johansson, and Matthieu Barreau. "Iterative training of physics-informed neural networks with Fourier-enhanced features." The Fourteenth International Conference on Learning Representations.
2 comments
7qjS response, pt. 2
W2 (periodic benchmarks, appendix baselines): The periodic setting is equally favorable to the baselines, since JAX-PI and PirateNet are themselves Fourier-feature methods and their released configurations for all four problems enforce periodicity exactly. App. B shows that the method carries over to non-periodic, irregular domains by embedding them in a periodic box. To isolate the effect of the representation, we compare there against parameter-matched MLP and RFF baselines trained with the same protocol; beignet is 2.4–5.7× more accurate over three seeds. Its errors, (1.56 ± 0.26)×10⁻³ on Poisson2D-Classic and (7.22 ± 3.99)×10⁻⁴ on Poisson–Boltzmann, are also 7.9× and 20× lower than the best results reported in PINNacle (1.23×10⁻² and 1.43×10⁻², both with NTK loss weighting).
Q2 (ModifiedMLP, causal weighting): beignet in general does not strictly depend on these components. The Burgers experiment (Sec. 5.2), the irregular-domain problems (App. B) and image fitting (App. C) use neither. In the main benchmarks we keep them because the baselines use them, so that the comparison isolates the contribution of the Fourier feature embedding as the main contribution of our paper; in general we see them as orthogonal components which are helpful in their own right.
Q3 (high-frequency benchmarks, RFF baselines): The baselines in the main benchmark are already RFF methods. For each, we replace the RFF embedding of JAX-PI with our pyramid. The benchmarks are also high-frequency in the sense that matters at high precision. Representing the reference solutions to 10⁻⁶ relative error requires spatial modes up to k ≈ 85 for KdV and k ≈ 250 for Allen–Cahn. In contrast, the baselines' Fourier features carry no energy above about k = 12–17 at that level, so every higher mode must be produced by the MLP. [1] is earlier work from the same group as JAX-PI and PirateNet, whose later architectures (which incorporate tuned RFF) we compare against directly in the paper. [2] proposes an iterative training procedure that is complementary to the choice of embedding. We are happy to further discuss both in related work.
Q4 (temporal blending): u_t can jump at the interior knots, but the effect is small for the first-order-in-time problems we study. We checked this on the Allen–Cahn and KdV models from Table 1 (seed 42), which each have 6 interior knots. At every knot we evaluated the exact one-sided limits of u_t on the 512-point reference grid, and measured the jump as ‖u_t⁺ − u_t⁻‖₂/‖u_t‖₂ over space. We also compared the PDE residual RMS over space at the knots with that at the midpoints between knots. Values are mean ± standard deviation over the knots (both one-sided limits for the residual) or over the midpoints:
| Allen–Cahn | KdV | |
|---|---|---|
| relative jump in u_t | (2.06 ± 0.65)×10⁻⁴ | (4.41 ± 1.46)×10⁻⁴ |
| residual RMS at the knots | (1.24 ± 0.31)×10⁻⁴ | (9.29 ± 2.42)×10⁻⁴ |
| residual RMS at the midpoints | (2.58 ± 0.67)×10⁻⁵ | (2.65 ± 0.63)×10⁻⁴ |
The jump in u_t is below 10⁻³ of ‖u_t‖, and the residual is locally elevated at the knots. Even with the knots, beignet reaches a relative L2 error of (1.88 ± 0.22)×10⁻⁶ on Allen–Cahn and (3.71 ± 0.63)×10⁻⁵ on KdV (Table 1), 6.4× and 7.0× lower than the strongest baseline, PirateNet. For PDEs that are second order in time, such as the wave equation, the linear interpolation could easily be replaced by e.g. spline interpolation of the same anchors, which would leave the spatial FFT path and the parameter count unchanged. We are happy to note this as a future direction.
W3 (training hyperparameters): These can be found in the supplement (Table 10, App. H) and in the code release (reproducibility statement).
7qjS response, pt. 1
Thank you for the review. Please find our responses below.
W1 (theory): The NTK analysis in [1] can be repeated for beignet by substituting the pyramid's eigenspectrum for that of the Gaussian frequency matrix. With a linear readout, each level contributes (1/N_ℓ) Σ_{|k|<N_ℓ/2} e^{2πik(x−x′)}. The NTK is therefore diagonal in Fourier modes, with a staircase spectrum λ(k) = Σ_{ℓ: |k|<N_ℓ/2} 1/N_ℓ that is exactly zero beyond the finest level's Nyquist frequency. Gaussian RFF instead has a Gaussian spectrum, with σ setting both bandwidth and decay. For direct regression, NTK theory guarantees convergence. In the infinite-width limit the kernel stays fixed during gradient descent, so training converges to the kernel regression solution and learns each eigenfunction at a rate set by its eigenvalue (Jacot et al., 2018; Lee et al., 2019), which [1] uses to explain how σ shapes spectral bias. In this regression setting, the pyramid's wider spectral coverage is favorable, because with a linear readout its eigenvalues decay only polynomially (about |k|^(−d)) up to the finest level's Nyquist frequency, whereas the expected Gaussian RFF spectrum decays as exp(−|k|²/2σ²), so high-frequency modes are learned more slowly. This is consistent with direct image fitting (App. C, the regression setting of Tancik et al., 2020), where beignet reaches a PSNR of 58.4 dB vs. 36.7 dB for RFF (σ = 10) after the same 2000 Adam steps. We are happy to add the derivation to the paper. However, this guarantee does not extend to nonlinear PDEs in general. There, the NTK changes during training even in the infinite-width limit (Bonfanti et al., NeurIPS 2024), whereas the PINN analysis in [1] assumes a frozen kernel. To our knowledge, existing convergence results for PINN training cover only linear PDEs or restricted classes of second-order semilinear PDEs, which exclude the equations we study, for RFF and for our method alike. App. G gives an empirical, operator-composed modal NTK, but we use it as a diagnostic for stiffness rather than as a theoretical guarantee.
Q1 (hyperparameter choice): Since the NTK guarantees do not hold for the nonlinear PDEs we study, the baseline's RFF bandwidth σ is also tuned per PDE in practice, with σ = 2 for Allen–Cahn and σ = 1 for the other three problems in the released JAX-PI and PirateNet configurations. Similarly, we choose levels and channels with a small parameter-matched sweep (Sec. 5.3), with the finest level capped by the resolution the solution requires. The feature grids are initialized uniformly at random in [−0.1, 0.1] on every level, and the decoder uses the same initialization as the JAX-PI baselines (Table 10); on Ginzburg–Landau and Gray–Scott we additionally fit the decoder's output layer to the initial condition by least squares, as PirateNet does. We have also found that poorly chosen configurations can be identified without a reference solution, using a standard a posteriori error estimate (Hairer, Nørsett & Wanner, Solving Ordinary Differential Equations I, 1993). We propagate the model's own residual and initial-condition error through the PDE, linearized about the model's solution, with a single linear solve. On the configurations of Fig. 2 (relative L2 error):
On both PDEs the estimate ranks the configurations consistently with the true error (Spearman ρ = 0.93 on Allen–Cahn, 1.00 on KdV). It flags the poor configurations, i.e. the 3–5-level Allen–Cahn models (18–50× above the best error) and the 7–10-level KdV models (error 10⁻² or larger), and it selects a configuration within 1.35× of the best error on Allen–Cahn and the best one on KdV.
References
Jacot et al. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS 2018.
Lee et al. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS 2019.
Tancik et al. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS 2020.
Bonfanti et al. The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks. NeurIPS 2024.
Hairer, Nørsett & Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, 1993.
(Continued in the next comment.)