acceptodds
Under review as a conference paper at ICLR 2027

Fourier Feature Pyramids for Physics-Informed Neural Networks

Abstract

We present an improved neural field architecture for solving partial differential equations (PDEs). Current physics-informed neural networks (PINNs) provide a flexible framework for solving PDEs, but they struggle to achieve highly accurate solutions and require computation that scales poorly with parameter count. Our model, which we call (Bandlimited Embedding with Interpolated Grid Network), replaces the random Fourier feature embedding used by existing PINN models with a trainable multi-resolution Fourier feature pyramid. To query at a continuous coordinate, we use Fourier interpolation at each level of the pyramid to return features at the input coordinate, and then decode this vector with a fully-connected neural network trunk. Our model provides multiple benefits: 1) Spatial derivatives can be computed efficiently by using the chain rule to compose derivatives of the neural network computed with automatic differentiation with derivatives of the feature grid computed spectrally by the Fast Fourier transform (FFT). 2) can achieve higher accuracy in a compute-efficient manner by scaling the parameter count of this Fourier feature pyramid, instead of the less-efficient strategy of scaling the neural network architecture. 3) can directly control the representation bandlimit, resulting in more stable optimization for difficult PDEs. We demonstrate that finds significantly more accurate solutions on PDE benchmarks using fewer parameters than state-of-the-art PINN methods. We further evaluate on the self-similar inviscid Burgers blowup problem and show that it can minimize residuals to near machine precision using Adam, an accuracy regime previously attained only by using computationally expensive higher-order optimizers.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PINNs, Physics, PDEs, Neural Networks, Machine Learning, Fourier Features

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7qjSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an improved PINNs architecture, beignet, which targets the difficulties of standard PINNs: struggle to achieve highly accurate solutions and their computation scales poorly with parameter count. Beignet method is based on the random Fourier feature (RFF) input embedding method in PINNs, which replaces it with a learned multi-resolution feature pyramid. It uses Fourier interpolation at each level of the pyramid to embed the input coordinates and send the features into the network. By directly controlling the representation bandlimit, the method addresses the drawback of RFF. Experiments show that it consistently outperforms several SOTA methods on multiple challenging benchmarks.

Critical strengths
  1. This paper addresses the drawback of RFF in PINNs, where RFF is a standard tool for PINNs for mitigating the spectral bias problems.
  2. This paper has a clear presentation and motivation, and the numerical results show the effectiveness of the method on multiple benchmarks.
  3. This paper provides a method that can be easily combined with existing methods and is effective for a wide range of problems.
Critical weaknesses
  1. This paper lacks theoretical analysis and spectral-bias benchmarks. The abstract/intro/background of the paper emphasizes spectral bias and the bandlimit drawback of RFF, and introduces the NTK in the background section. However, the rest of the paper does not analyze the NTK of the proposed method theoretically. It also provides no other theoretical results supporting the method's effectiveness, and few popular spectral-bias / high-frequency benchmarks.
  2. The main benchmarks are mostly periodic boundary problems, such as Allen–Cahn and KdV, which suit the structure of the method particularly well from a theoretical view. Even experiments on other benchmarks are included in later parts, but the baselines chosen for comparison are not SOTA for those benchmarks.
  3. The paper only states that it follows the JAX-PI training procedure, without specifying the training hyperparameters, data, and other detailed settings. JAX-PI is already an advanced structure combined with advanced training techniques.
Questions for authors
  1. For RFF-PINNs, the bandwidth is usually chosen according to the frequency domain of the given PDE, which has some theoretical guarantees based on the NTK theory. Could the authors explain how the hyperparameters of beignet are chosen, e.g., the number of levels LL and the parameter initialization?
  2. The paper states that they use techniques such as ModifiedMLP and causal time-chunk weighting based on JAX-PI training. Can the author ensure that the advantage of beignet persists when these components are removed? Can beignet stand as a relatively independent algorithm, or must it be used together with these methods?
  3. The choice of benchmarks and baselines could be better aligned with spectral bias and improvement on RFF methods if the author wants to emphasize it. For the benchmarks, could the authors try some popular high-frequency problems such as high-frequency heat equations or maybe high-frequency Helmholtz equations? For the baselines, the RFF methods should be compared more, rather than mostly in appendix, such as [1] and [2]. 4.In the Temporal Pyramid Blending section, θ(t)\theta(t) is piecewise linear in time. Would this property make the method unsuitable for equations involving second- or higher-order time derivatives? E.g., in the wave equation.

[1] Wang, Sifan, Hanwen Wang, and Paris Perdikaris. "On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks." Computer Methods in Applied Mechanics and Engineering 384 (2021): 113938.

[2] Wu, Yulun, Aguiar, M., Karl H. Johansson, and Matthieu Barreau. "Iterative training of physics-informed neural networks with Fourier-enhanced features." The Fourteenth International Conference on Learning Representations.

2 comments
AuthorsOpenReview ↗

7qjS response, pt. 1

Thank you for the review. Please find our responses below.

W1 (theory): The NTK analysis in [1] can be repeated for beignet by substituting the pyramid's eigenspectrum for that of the Gaussian frequency matrix. With a linear readout, each level contributes (1/N_ℓ) Σ_{|k|<N_ℓ/2} e^{2πik(x−x′)}. The NTK is therefore diagonal in Fourier modes, with a staircase spectrum λ(k) = Σ_{ℓ: |k|<N_ℓ/2} 1/N_ℓ that is exactly zero beyond the finest level's Nyquist frequency. Gaussian RFF instead has a Gaussian spectrum, with σ setting both bandwidth and decay. For direct regression, NTK theory guarantees convergence. In the infinite-width limit the kernel stays fixed during gradient descent, so training converges to the kernel regression solution and learns each eigenfunction at a rate set by its eigenvalue (Jacot et al., 2018; Lee et al., 2019), which [1] uses to explain how σ shapes spectral bias. In this regression setting, the pyramid's wider spectral coverage is favorable, because with a linear readout its eigenvalues decay only polynomially (about |k|^(−d)) up to the finest level's Nyquist frequency, whereas the expected Gaussian RFF spectrum decays as exp(−|k|²/2σ²), so high-frequency modes are learned more slowly. This is consistent with direct image fitting (App. C, the regression setting of Tancik et al., 2020), where beignet reaches a PSNR of 58.4 dB vs. 36.7 dB for RFF (σ = 10) after the same 2000 Adam steps. We are happy to add the derivation to the paper. However, this guarantee does not extend to nonlinear PDEs in general. There, the NTK changes during training even in the infinite-width limit (Bonfanti et al., NeurIPS 2024), whereas the PINN analysis in [1] assumes a frozen kernel. To our knowledge, existing convergence results for PINN training cover only linear PDEs or restricted classes of second-order semilinear PDEs, which exclude the equations we study, for RFF and for our method alike. App. G gives an empirical, operator-composed modal NTK, but we use it as a diagnostic for stiffness rather than as a theoretical guarantee.

Q1 (hyperparameter choice): Since the NTK guarantees do not hold for the nonlinear PDEs we study, the baseline's RFF bandwidth σ is also tuned per PDE in practice, with σ = 2 for Allen–Cahn and σ = 1 for the other three problems in the released JAX-PI and PirateNet configurations. Similarly, we choose levels and channels with a small parameter-matched sweep (Sec. 5.3), with the finest level capped by the resolution the solution requires. The feature grids are initialized uniformly at random in [−0.1, 0.1] on every level, and the decoder uses the same initialization as the JAX-PI baselines (Table 10); on Ginzburg–Landau and Gray–Scott we additionally fit the decoder's output layer to the initial condition by least squares, as PirateNet does. We have also found that poorly chosen configurations can be identified without a reference solution, using a standard a posteriori error estimate (Hairer, Nørsett & Wanner, Solving Ordinary Differential Equations I, 1993). We propagate the model's own residual and initial-condition error through the PDE, linearized about the model's solution, with a single linear solve. On the configurations of Fig. 2 (relative L2 error):

Levels34 (KdV best)56789 (Allen–Cahn best)10
Allen–Cahn, true error7.8e-55.3e-52.8e-52.6e-62.1e-61.6e-61.5e-62.9e-6
Allen–Cahn, estimate (no reference)7.8e-55.3e-52.7e-51.5e-61.0e-61.4e-61.1e-62.5e-6
KdV, true error9.8e-53.2e-57.9e-51.9e-41.3e-25.9e-16.9e-17.8e-1
KdV, estimate (no reference)9.8e-53.2e-57.8e-51.9e-41.3e-26.5e-19.3e-11.9e2

On both PDEs the estimate ranks the configurations consistently with the true error (Spearman ρ = 0.93 on Allen–Cahn, 1.00 on KdV). It flags the poor configurations, i.e. the 3–5-level Allen–Cahn models (18–50× above the best error) and the 7–10-level KdV models (error 10⁻² or larger), and it selects a configuration within 1.35× of the best error on Allen–Cahn and the best one on KdV.

References

Jacot et al. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS 2018.

Lee et al. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS 2019.

Tancik et al. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS 2020.

Bonfanti et al. The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks. NeurIPS 2024.

Hairer, Nørsett & Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, 1993.

(Continued in the next comment.)

AuthorsOpenReview ↗

7qjS response, pt. 2

W2 (periodic benchmarks, appendix baselines): The periodic setting is equally favorable to the baselines, since JAX-PI and PirateNet are themselves Fourier-feature methods and their released configurations for all four problems enforce periodicity exactly. App. B shows that the method carries over to non-periodic, irregular domains by embedding them in a periodic box. To isolate the effect of the representation, we compare there against parameter-matched MLP and RFF baselines trained with the same protocol; beignet is 2.4–5.7× more accurate over three seeds. Its errors, (1.56 ± 0.26)×10⁻³ on Poisson2D-Classic and (7.22 ± 3.99)×10⁻⁴ on Poisson–Boltzmann, are also 7.9× and 20× lower than the best results reported in PINNacle (1.23×10⁻² and 1.43×10⁻², both with NTK loss weighting).

Q2 (ModifiedMLP, causal weighting): beignet in general does not strictly depend on these components. The Burgers experiment (Sec. 5.2), the irregular-domain problems (App. B) and image fitting (App. C) use neither. In the main benchmarks we keep them because the baselines use them, so that the comparison isolates the contribution of the Fourier feature embedding as the main contribution of our paper; in general we see them as orthogonal components which are helpful in their own right.

Q3 (high-frequency benchmarks, RFF baselines): The baselines in the main benchmark are already RFF methods. For each, we replace the RFF embedding of JAX-PI with our pyramid. The benchmarks are also high-frequency in the sense that matters at high precision. Representing the reference solutions to 10⁻⁶ relative error requires spatial modes up to k ≈ 85 for KdV and k ≈ 250 for Allen–Cahn. In contrast, the baselines' Fourier features carry no energy above about k = 12–17 at that level, so every higher mode must be produced by the MLP. [1] is earlier work from the same group as JAX-PI and PirateNet, whose later architectures (which incorporate tuned RFF) we compare against directly in the paper. [2] proposes an iterative training procedure that is complementary to the choice of embedding. We are happy to further discuss both in related work.

Q4 (temporal blending): u_t can jump at the interior knots, but the effect is small for the first-order-in-time problems we study. We checked this on the Allen–Cahn and KdV models from Table 1 (seed 42), which each have 6 interior knots. At every knot we evaluated the exact one-sided limits of u_t on the 512-point reference grid, and measured the jump as ‖u_t⁺ − u_t⁻‖₂/‖u_t‖₂ over space. We also compared the PDE residual RMS over space at the knots with that at the midpoints between knots. Values are mean ± standard deviation over the knots (both one-sided limits for the residual) or over the midpoints:

Allen–CahnKdV
relative jump in u_t(2.06 ± 0.65)×10⁻⁴(4.41 ± 1.46)×10⁻⁴
residual RMS at the knots(1.24 ± 0.31)×10⁻⁴(9.29 ± 2.42)×10⁻⁴
residual RMS at the midpoints(2.58 ± 0.67)×10⁻⁵(2.65 ± 0.63)×10⁻⁴

The jump in u_t is below 10⁻³ of ‖u_t‖, and the residual is locally elevated at the knots. Even with the knots, beignet reaches a relative L2 error of (1.88 ± 0.22)×10⁻⁶ on Allen–Cahn and (3.71 ± 0.63)×10⁻⁵ on KdV (Table 1), 6.4× and 7.0× lower than the strongest baseline, PirateNet. For PDEs that are second order in time, such as the wave equation, the linear interpolation could easily be replaced by e.g. spline interpolation of the same anchors, which would leave the spatial FFT path and the parameter count unchanged. We are happy to note this as a future direction.

W3 (training hyperparameters): These can be found in the supplement (Table 10, App. H) and in the code release (reproducibility statement).

Reviewer_hV3POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes beignet, a new PINN architecture based on a multi-resolution Fourier feature pyramid. The main idea is to represent different spatial frequencies using multiple trainable feature grids instead of random Fourier features. The method computes spatial derivatives using FFTs. The method is tested on four time-dependent PDE benchmarks and a self-similar Burgers problem. The results are strong and show improvements over JAX-PI and Piratenet on the tested problems. The paper also shows parameter efficiency and some ablations. However, most main experiments are on periodic structured domains, and the method requires PDE-specific choices of spectral resolution.

Critical strengths
  • The method improves over JAX-PI and PirateNet on all four main PDE benchmarks.
  • The paper includes ablations. In particular, the coarse-to-fine pyramid, temporal blending, and grid shifting are studied separately.
  • The FFT-based derivative computation gives a speed advantage compared with pointwise derivative evaluation.
Critical weaknesses
  • The novelty is limited in my opinion. Fourier features, bandlimited representations, and multi-resolution representations already exist. The main contribution is mostly how these ideas are organized together for PINNs.
  • The main comparison is mostly against JAX-PI and PirateNet. More direct comparisons with other spectral PINNs, deterministic Fourier bases, learnable Fourier features, or other recent high-accuracy PINN architectures would make the results more convincing.
  • The best pyramid configuration is PDE dependent. Allen-Cahn needs a much larger bandwidth while KdV works better with a smaller one. It is not clear how these choices would be made for a new PDE without trying several configurations.
  • Most of the main experiments use periodic PDEs on structured grids, which is a favorable setting for Fourier methods. The irregular non-periodic experiments are presented, but are limited to two relatively simple stationary problems.
  • The near machine-precision claim on the Burgers problem is based mainly on PDE residual values. A stronger comparison with a high-accuracy reference solution would help.
Questions for authors
  • Can the authors compare against more direct spectral baselines, such as deterministic Fourier features, learnable Fourier features, Frozen-PINNs, Fourier PINNs, or other recent frequency-based PINN methods?
  • How should the number of pyramid levels and maximum frequency be selected for a new PDE? Can this be done without access to an accurate reference solution?
  • Can the authors provide more experiments on non-periodic domains, irregular geometries, and higher-dimensional PDEs to show that the advantage is not mainly from the FFT-friendly setting?
  • How does the memory and parameter cost scale in 3D, where storing high-resolution feature grids can become expensive?
  • For the self-similar Burgers experiment, can the authors provide direct solution error against a high-accuracy numerical reference, in addition to the very small PDE residual?
3 comments
AuthorsOpenReview ↗

hV3P response, pt. 1

Thank you for the review. Please find our responses below.

W1 (novelty): We agree that Fourier representations, bandlimiting and multiresolution are classical, and we do not claim them individually. Our contribution is a PINN representation that combines a trainable, explicitly band-limited, multi-resolution Fourier feature pyramid with continuous Fourier interpolation, FFT differentiation, and stochastic grid shifts. The ablations show that each component matters. At matched parameter count, the pyramid reaches 87× lower error than a single-resolution grid (Table 6). On Allen–Cahn, stochastic grid shifts are the difference between a relative L2 error of 0.95 and 1.5×10⁻⁶ (Table 5), and temporal blending lowers the error a further 2.8× (Table 7). The novelty is not Fourier analysis itself, but that this particular combination of classical spectral structure with a neural decoder yields a notably effective high-precision PINN representation, which improves over JAX-PI and PirateNet on all four main benchmarks.

W2/Q1 (Fourier feature baselines): JAX-PI and PirateNet both use Gaussian random Fourier feature embeddings, whose frequency matrix is also trained (learnable) in the released code. They are the strongest published baselines tuned specifically for this benchmark, which is why we selected them. The PirateNet paper already compares against earlier PINN variants on Allen–Cahn and KdV (e.g. causal training, time marching, self-attention, RBA-PINNs) and improves on all of them, and both methods have publicly released code and tuned configurations for these problems. By Eq. 5, the fixed-resolution beignet model in the App. D.2 ablation reduces to a deterministic Fourier feature embedding with a learned linear map. The pyramid outperforms it by 87× in error. Other spectral PINNs, such as Frozen-PINNs and Fourier PINNs, have not been tuned or reported on these benchmarks, so rather than re-tuning each of them ourselves, we compare against the strongest published configurations for these problems.

W3/Q2 (PDE-dependent configuration): The optimal pyramid allocation is indeed PDE-dependent, but this is shared by everything we compare against on these benchmarks. The released JAX-PI and PirateNet configurations vary several hyperparameters across equations, and the reference solutions come from numerical solvers whose settings and resolution are chosen per PDE in practice. Our pyramid introduces a small set of interpretable choices (maximum resolution, number of levels), and performance remains good even for suboptimal choices (Fig. 2).

Poorly chosen configurations can also be identified without a reference solution, using a standard a posteriori error estimate (Hairer, Nørsett & Wanner, Solving Ordinary Differential Equations I, 1993). We propagate the model's own residual and initial-condition error through the PDE, linearized about the model's solution, with a single linear solve. On the configurations of Fig. 2 (relative L2 error):

Levels34 (KdV best)56789 (Allen–Cahn best)10
Allen–Cahn, true error7.8e-55.3e-52.8e-52.6e-62.1e-61.6e-61.5e-62.9e-6
Allen–Cahn, estimate (no reference)7.8e-55.3e-52.7e-51.5e-61.0e-61.4e-61.1e-62.5e-6
KdV, true error9.8e-53.2e-57.9e-51.9e-41.3e-25.9e-16.9e-17.8e-1
KdV, estimate (no reference)9.8e-53.2e-57.8e-51.9e-41.3e-26.5e-19.3e-11.9e2

On both PDEs the estimate ranks the configurations consistently with the true error (Spearman ρ = 0.93 on Allen–Cahn, 1.00 on KdV). It flags the poor configurations, i.e. the 3–5-level Allen–Cahn models (18–50× above the best error) and the 7–10-level KdV models (error 10⁻² or larger), and it selects a configuration within 1.35× of the best error on Allen–Cahn and the best one on KdV.

(Continued in the next comment.)

AuthorsOpenReview ↗

hV3P response, pt. 2

W4/Q3 (periodic domains): The periodic setting is equally favorable to the baselines. JAX-PI and PirateNet are themselves Fourier-feature methods, and on all four problems their released configurations also enforce periodicity exactly. Beyond it, App. B handles non-periodic irregular domains by embedding them in a periodic box, where beignet beats parameter-matched MLP and RFF baselines by 2.4–5.7× over three seeds. Its errors there, (1.56 ± 0.26)×10⁻³ on Poisson2D-Classic and (7.22 ± 3.99)×10⁻⁴ on Poisson–Boltzmann, are also 7.9× and 20× lower than the best results reported in PINNacle (1.23×10⁻² and 1.43×10⁻², both with NTK loss weighting). The Burgers problem is posed on an unbounded domain. Altogether, the paper evaluates beignet on seven PDEs, comprising four time-dependent periodic benchmarks (including third-order dispersion and coupled reaction–diffusion systems), two non-periodic problems on irregular domains, and a self-similar profile on an unbounded domain. We believe this already demonstrates the versatility of the method beyond the FFT-friendly setting, and we leave very high-dimensional PDEs to future work.

W5/Q5 (Burgers solution error): For λ = 1/2 the profile is known in implicit form, U³ + U + y = 0, so no numerical reference is needed. We compare against the exact profile in 60-digit arithmetic, with the same far-field normalization that our model and the BFGS reference impose, on a uniform grid of 10⁵ points, 100× denser than the paper's grid ("dense"), and on 10⁵ held-out random points ("random"), covering y ∈ [10⁻⁴, 5×10¹²]. The comparison showed that the 20k-step budget in Table 2 left our model optimization-limited. At a matched budget of 60k steps, MLP + Adam and the BFGS reference keep their Table 2 settings, and our model uses Adam with lr 3×10⁻⁵ (mean ± standard deviation over 3 seeds):

max |U − U_exact| (dense)max |U − U_exact| (random)max |U_y − U_y,exact| (dense)max |U_y − U_y,exact| (random)residual MSE (dense)residual MSE (random)
MLP + Adam(1.25 ± 1.67)×10⁻³(1.25 ± 1.67)×10⁻³(8.17 ± 4.28)×10⁻⁴(8.17 ± 4.28)×10⁻⁴(3.53 ± 5.88)×10⁻⁷(3.55 ± 5.91)×10⁻⁷
MLP + BFGS(1.90 ± 0.89)×10⁻¹⁰(1.90 ± 0.89)×10⁻¹⁰(6.11 ± 4.66)×10⁻¹⁰(6.11 ± 4.66)×10⁻¹⁰(3.56 ± 2.40)×10⁻²⁰(3.56 ± 2.40)×10⁻²⁰
beignet + Adam(2.42 ± 1.62)×10⁻¹⁰(2.42 ± 1.62)×10⁻¹⁰(3.95 ± 2.72)×10⁻⁸(3.95 ± 2.72)×10⁻⁸(1.85 ± 1.94)×10⁻²⁰(1.84 ± 1.93)×10⁻²⁰

For every model, the dense and random test points give errors in U and U_y that agree to three significant digits, and residual MSEs that agree to two. With Adam alone, beignet matches the BFGS solution in the maximum error of U and in the residual. BFGS remains more accurate in U_y near the origin. We will add these solution errors to Table 2.

Q4 (3D memory): A dyadic pyramid stores about (8/7)·N³·C values per time slice, so its cost is dominated by the finest level, i.e. by a single grid at the resolution the solution requires. To check feasibility, we trained a toy stationary 3D Poisson problem with our FFT feature path on a single 80 GB A100. A pyramid up to 256³ with 4 channels per level (77M parameters) uses 21.9 GiB of peak memory at 0.51 s per step, and 8 channels use 43.6 GiB. 512³ grids exceed a single A100 GPU and would require factorized or sparse grids, which we leave to future work.

Thanks for the response and additional experiments, especially on hyperparameter selection, Burgers and 3D memory usage. However, I still think my main concerns remain. The ablation results show that the proposed method works well, but most of the techniques used are already known, so the novelty is still limited. Also, comparisons with other closely related spectral methods are missing. Most of the strong results are still on periodic problems, so its not clear how well it works on more general problems. Hence, I keep my original score.

Reviewer_hwC3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces beignet, which replaces random Fourier embeddings with trainable Fourier feature grids at multiple resolutions. A shared MLP decoder maps these features to a continuous field. Spatial feature derivatives are computed in Fourier space and passed through the decoder using the chain rule and forward-mode automatic differentiation. Structured grids allow the FFT computation to be reused. The method also uses temporal pyramid blending and random grid shifts. On four JAX-PI PDE tasks, the authors report lower relative L2 error than JAX-PI and PirateNet over three random seeds. The appendices add non-periodic problems on irregular domains, ablations, and timing results. A self-similar Burgers experiment shows that Adam can reach a very small residual on a diagnostic grid.

Critical strengths
  1. The method improves representation and computation. The feature pyramid controls capacity across frequency bands, and FFT-based derivatives speed up residual evaluation. The matched-NFE comparisons in Appendix E support the computational gains.

  2. The accuracy gains are strong and supported by ablations. Across four PDEs, several errors are about an order of magnitude below PirateNet. The grid-shift and parameter-matched ablations show that both sampling and pyramid structure contribute to the gains.

Critical weaknesses
  1. The Burgers experiment reports residual accuracy, not solution accuracy. A very small squared residual does not establish a solution near machine precision. The paper should report errors in the profile and its derivatives against an independent reference, with denser test points.

  2. The closest baselines and full training costs are missing. The main comparisons do not include a closely related Fourier-basis method. Timing focuses on raw training steps for Allen–Cahn. Time to a target error and peak memory on more tasks would better support the efficiency claim.

Questions for authors
  1. For fixed λ = 0.5, could you report independent errors in U and U_y against the implicit solution with the same far-field normalization? Could you also check a denser grid, random test points, and residuals recomputed at higher precision? Does “near machine precision” in Table 2 refer to squared residual, maximum residual, or solution accuracy?

  2. The temporal blending in Eq. (7) is generally only C0, so time derivatives can jump at the knots. Passing raw t to the decoder does not automatically remove these jumps. Have you measured u_t and the PDE residual on both sides of the knots? This does not necessarily invalidate a weak-solution approximation for these first-order-in-time PDEs, but the smoothness limits should be stated. Similarly, a bandlimited feature field g generally loses its strict bandlimit after a nonlinear tanh or swish decoder. Should the spectral-control claim be limited explicitly to the features?

  3. Could you add a closely related Fourier-basis baseline, or compare a single-level trainable Fourier expansion with the pyramid under the same training setup? Reporting time to a target error and peak memory on at least one 2D task would also strengthen the main claim more than adding further PDE names.

2 comments
AuthorsOpenReview ↗

hwC3 response, pt. 1

Thank you for the review. Please find our responses below.

W1/Q1 (Burgers solution error): For λ = 1/2 the profile is known in implicit form: U³ + U + y = 0, with U_y = −1/(1 + 3U²).

Errors in U and U_y, denser grid and random test points. Our model and the BFGS reference fix the scaling through the same far-field condition, so we compare against the exact profile with that normalization, evaluated in 60-digit arithmetic. We evaluate on a uniform grid of 10⁵ points, 100× denser than the paper's grid ("dense"), and on 10⁵ held-out random points ("random"), covering y ∈ [10⁻⁴, 5×10¹²]. This showed that the 20k-step budget in Table 2 left our model optimization-limited, with its error concentrated in the slope at y = 0. At a matched budget of 60k steps (mean ± standard deviation over 3 seeds):

max |U − U_exact| (dense)max |U − U_exact| (random)max |U_y − U_y,exact| (dense)max |U_y − U_y,exact| (random)residual MSE (dense)residual MSE (random)
MLP + Adam(1.25 ± 1.67)×10⁻³(1.25 ± 1.67)×10⁻³(8.17 ± 4.28)×10⁻⁴(8.17 ± 4.28)×10⁻⁴(3.53 ± 5.88)×10⁻⁷(3.55 ± 5.91)×10⁻⁷
MLP + BFGS(1.90 ± 0.89)×10⁻¹⁰(1.90 ± 0.89)×10⁻¹⁰(6.11 ± 4.66)×10⁻¹⁰(6.11 ± 4.66)×10⁻¹⁰(3.56 ± 2.40)×10⁻²⁰(3.56 ± 2.40)×10⁻²⁰
beignet + Adam(2.42 ± 1.62)×10⁻¹⁰(2.42 ± 1.62)×10⁻¹⁰(3.95 ± 2.72)×10⁻⁸(3.95 ± 2.72)×10⁻⁸(1.85 ± 1.94)×10⁻²⁰(1.84 ± 1.93)×10⁻²⁰

For every model, the dense and random test points give errors in U and U_y that agree to three significant digits, and residual MSEs that agree to two. With Adam alone, beignet matches the BFGS solution in the maximum error of U and in the residual. BFGS remains more accurate in U_y near the origin.

Higher-precision residuals. Recomputing the residuals in 60-digit arithmetic changes them by less than 3×10⁻¹².

"Near machine precision". This refers to the residual MSE. We will update Table 2 to the 60k budget, add these solution errors, and are happy to reword the claim accordingly.

Q2 (temporal knots): u_t can jump at the interior knots, but the effect is small in practice. We checked this on the Allen–Cahn and KdV models from Table 1 (seed 42), which each have 6 interior knots. At every knot we evaluated the exact one-sided limits of u_t on the 512-point reference grid, and measured the jump as ‖u_t⁺ − u_t⁻‖₂/‖u_t‖₂ over space. We also compared the PDE residual RMS over space at the knots with that at the midpoints between knots. Values are mean ± standard deviation over the knots (both one-sided limits for the residual) or over the midpoints:

Allen–CahnKdV
relative jump in u_t(2.06 ± 0.65)×10⁻⁴(4.41 ± 1.46)×10⁻⁴
residual RMS at the knots(1.24 ± 0.31)×10⁻⁴(9.29 ± 2.42)×10⁻⁴
residual RMS at the midpoints(2.58 ± 0.67)×10⁻⁵(2.65 ± 0.63)×10⁻⁴

The jump in u_t is below 10⁻³ of ‖u_t‖, and the residual is locally elevated at the knots. Even with the knots, beignet reaches a relative L2 error of (1.88 ± 0.22)×10⁻⁶ on Allen–Cahn and (3.71 ± 0.63)×10⁻⁵ on KdV (Table 1), 6.4× and 7.0× lower than the strongest baseline, PirateNet. Where higher temporal smoothness matters, e.g. for PDEs that are second order in time, the linear interpolation can be replaced by spline interpolation of the same anchors. We will note this limitation in the revision.

Q2 (bandlimit): We agree that the strict bandlimit applies to the features, not to the decoded output, and we will state this explicitly. As Tancik et al. (2020) observe for Fourier feature networks, a nonlinear decoder spreads the kernel spectrum around the embedded frequencies with a smooth falloff. In practice, this spreading decays strongly beyond the feature bandlimit. The output modal NTK of the KdV models at initialization (Fig. 10, "output" curves) measures how strongly gradient descent can change each output Fourier mode. For the 4-level model, whose features stop at |k| = 8, it falls by a factor of 34 from |k| = 8 to |k| = 16, 2×10⁴ by |k| = 32, and 10⁹ by |k| = 64. For the 6-level model (features up to |k| = 32), it falls by 57× at |k| = 64 and 4×10⁴ at |k| = 127. So although the bandlimit on the output is not strict, it has a significant effect on which frequencies are learned, as modes well above it are learned much more slowly.

(Continued in the next comment.)

AuthorsOpenReview ↗

hwC3 response, pt. 2

W2/Q3 (Fourier feature baselines): Both main baselines (JAX-PI, PirateNet) are trainable Fourier-feature methods. We choose to compare against these Fourier-feature methods in particular as they are the strongest published baselines on this benchmark. The PirateNet paper already compares against earlier PINN variants on Allen–Cahn and KdV (e.g. causal training, time marching, self-attention, RBA-PINNs) and improves on all of them, and both JAX-PI and PirateNet have publicly released code and tuned configurations for these problems. The parameter-matched ablation in App. D.2 (Table 6), with all levels at the finest resolution, corresponds mathematically to a single-resolution, multi-channel trainable Fourier grid. The pyramid reaches 87× lower error than it.

W2 (timing): We agree that time-to-target error would be informative, but matched-NFE convergence curves on all four PDEs need more training than the rebuttal period allows. Instead, we extended the supplement's matched-NFE timing and peak-memory comparison to all four PDEs against nine-block PirateNet, and will clarify that our efficiency claim is per update.

Existing AC time-to-error results. For Allen–Cahn, the Table 1 runs provide a comparison at their native training configurations. Including compilation, evaluation, loss-weight updates and checkpointing, beignet reaches relative L2 error 1.2×10⁻⁵ after ~2.1 h, versus ~4.7 h for nine-block PirateNet (the target is reached by two of three PirateNet seeds). Beignet reaches 5×10⁻⁶ after ~2.6 h, a target neither baseline reaches within the reported training budget.

Matched-NFE throughput and peak memory across all four main PDEs. We extended the App. E comparison to Allen–Cahn, KdV, Ginzburg–Landau and Gray–Scott using the reference beignet FFT path and nine-block, width-256 PirateNet baselines. On the 15 successfully measured pairs, beignet is 4.0–11.3× faster per optimizer update and has substantially lower peak JAX allocation in every case.

Each run uses one NVIDIA A100 80 GB GPU, FP32 and highest matmul precision. We time five trials of 100 raw optimizer updates after compilation and ten warmup updates. The timed region excludes external sampler generation, evaluation, logging, checkpointing and periodic loss-weight updates. These rebuttal experiments were run on a different machine with the same GPU as the results originally reported in the supplement, so we reran the Allen-Cahn experiment on the new machine for fairness.

Raw optimizer time: seconds per 100 updates, mean ± sample standard deviation over five trials.

PDEMethod8,192 NFE16,384 NFE32,768 NFE65,536 NFE
Allen–Cahnbeignet1.949 ± 0.0032.857 ± 0.0034.424 ± 0.0046.461 ± 0.035
Allen–CahnPirateNet13.441 ± 0.00619.489 ± 0.00731.668 ± 0.03249.494 ± 0.149
KdVbeignet3.408 ± 0.0014.379 ± 0.0047.606 ± 0.00213.949 ± 0.015
KdVPirateNet15.674 ± 0.01922.071 ± 0.00737.006 ± 0.13759.673 ± 0.057
Ginzburg–Landaubeignet6.884 ± 0.0077.637 ± 0.0059.439 ± 0.00511.813 ± 0.009
Ginzburg–LandauPirateNet31.450 ± 0.04347.846 ± 0.03677.999 ± 0.116133.070 ± 0.093
Gray–Scottbeignet6.899 ± 0.0097.622 ± 0.0059.430 ± 0.00911.817 ± 0.011
Gray–ScottPirateNet27.842 ± 0.02143.018 ± 0.02670.475 ± 0.023N/A (OOM)

Peak JAX-allocated GPU memory (GiB). Peak memory is JAX's high-water mark in a fresh process (JIT, warmup and one 100-update window), excluding allocator-pool reservation and CUDA context.

PDEMethod8,192 NFE16,384 NFE32,768 NFE65,536 NFE
Allen–Cahnbeignet0.81.21.63.6
Allen–CahnPirateNet4.87.515.631.5
KdVbeignet1.11.32.85.3
KdVPirateNet4.68.918.536.8
Ginzburg–Landaubeignet2.92.93.13.8
Ginzburg–LandauPirateNet14.222.841.275.2
Gray–Scottbeignet2.92.93.13.8
Gray–ScottPirateNet14.223.045.5N/A (OOM)

N/A: Gray–Scott/PirateNet runs out of memory at 65,536 NFE on a single GPU.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.