acceptodds
Under review as a conference paper at ICLR 2027

Implicit Legendre Representation for Physics-informed Neural Network

Abstract

Physics-informed neural networks (PINNs) commonly employ Fourier-based representations to more accurately approximate high-frequency components in solutions to partial differential equations (PDEs). In this work, we challenge the need for spectral priors in representation design and propose Legendre representation networks (LERENs), which integrate learnable Legendre expansions into multilayer perceptrons (MLPs). We evaluate LERENs on five challenging PDE problems involving high-frequency components, complex solution structures, and high-dimensional domains. Experimental results demonstrate that LERENs outperform state-of-the-art Fourier-based representation methods on the evaluated benchmarks, establishing Legendre-based representations as a competitive alternative to Fourier-based representations in PINNs. The proposed representation paradigm may extend to other bases used in spectral methods, allowing their favorable mathematical properties to inform the design of neural representations.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
physics-informed neural network, implicit neural representation, spectral method

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9t7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Legendre Representation Networks (LERENs) to improve the representational capacity of physics-informed neural networks for high-frequency and structurally complex PDE solutions. Existing approaches for mitigating spectral bias in PINNs often rely on Fourier embeddings, periodic input encodings, or sinusoidal activations. LEREN offers an alternative by incorporating learnable Legendre polynomial expansions into the hidden layers of an MLP. Through multilayer composition, lower-degree polynomial components can generate higher-degree modes, which provides a non-Fourier mechanism for representing high-frequency structures. The method is evaluated on five PDE benchmarks, covering both sinusoidal or periodic problems and non-sinusoidal problems with high-dimensional or nonlinear dynamics. The reported results show that LEREN achieves the lowest errors among the compared methods across these benchmarks, demonstrating that Legendre-based representations can be competitive with existing Fourier-based approaches for PINNs.

Critical strengths
  1. The proposed method is simple, yet it delivers substantial empirical gains across multiple PDE benchmarks with diverse characteristics.

  2. The authors provide a relatively clean representation ablation that isolates the contribution of the Legendre expansion from parameter-count effects. This strengthens the evidence that the performance gains come from the proposed representation design.

Critical weaknesses

The universal approximation result has limited explanatory value. The universal approximation theorem is relatively weak in its current form. The proof effectively reduces LEREN to a standard tanh MLP by retaining only the first-order Legendre component. As a result, the theorem establishes expressivity, but it does not explain why the proposed Legendre representation should outperform conventional MLP-based PINNs in practice.

The optimization mechanism is not sufficiently understood. The paper shows that compositions of Legendre polynomials can generate higher-degree modes. However, this representational argument does not explain why gradient-based optimization can efficiently discover and exploit these modes during training. A deeper analysis of the optimization dynamics would strengthen the connection between the proposed representation and the observed empirical gains.

The central mechanism of higher-mode generation through composition is not directly validated. A central motivation of the paper is the following mechanism that depth leads to higher-mode generation. However, the experiments do not directly verify that deeper LEREN layers actually contain progressively higher-frequency or higher-degree components. The paper does not report the learned Legendre spectrum across layers, the Fourier spectrum of hidden features, the effective degree distribution, the learned coefficients cℓnc_{\ell n}, or the learned scaling parameters αℓ\alpha_\ell. At present, the main empirical evidence is essentially LEREN leads to lower final error. This result alone does not establish that the improvement comes from the proposed compositional mode-generation mechanism. A useful diagnostic would be to analyze the hidden representations at ℓ=1,2,3,4\ell = 1,2,3,4 and project them onto a Fourier basis. The resulting spectra could reveal whether higher-frequency content actually emerges as depth increases. A complementary analysis could track the learned Legendre coefficients and effective polynomial degrees across layers. The existing ablation results make this question particularly important. Increasing the maximum degree from 9→119 \rightarrow 11 degrades performance, and increasing the depth from four to five layers also increases the error. The paper summarizes this observation by stating that an appropriate depth and polynomial degree are required. However, these results may also indicate a more fundamental issue: higher-degree composition may become poorly conditioned. This possibility creates tension with the simple narrative that generating higher-degree modes is inherently beneficial. Further analysis of conditioning, optimization stability, or derivative magnitudes would help clarify this point.

The architectural choices appear highly heuristic, and their generality is unclear. The default LEREN architecture uses P0,…,P9P_0,\ldots,P_9 in the first layer, while the subsequent three layers use only P0,P1,P8,P9.P_0,P_1,P_8,P_9. This is a highly specific design choice. The paper does not provide a clear theoretical or empirical explanation for excluding P2,…,P7P_2,\ldots,P_7 from the later layers while retaining P8P_8 and P9P_9. It is also unclear why alternatives such as P1,P3,P5,P9P_1,P_3,P_5,P_9 or an adaptive basis-selection mechanism would be less effective. Most of the degree and depth ablations are conducted on the convection benchmark. This makes it difficult to determine whether the selected architecture generalizes across PDE families or reflects benchmark-specific tuning. Additional ablations across several PDEs would provide stronger evidence for the robustness of these architectural choices.

The experimental scope is still limited. The evaluation covers only five PDE benchmarks. These experiments include several useful settings, but the overall scope remains relatively narrow for establishing broad effectiveness across challenging PDE systems. At minimum, I would like to see an additional benchmark with complexity comparable to a two-dimensional Navier–Stokes problem. Such an experiment would provide stronger evidence that LEREN remains effective on more complex nonlinear and multiscale dynamics.

The experiments do not establish that the advantage is specific to the Legendre basis. The current results show that polynomial-enriched hidden representations can be effective, but they do not clearly establish that Legendre polynomials are the critical ingredient. A particularly important missing comparison is AC-PKAN [1], which uses a Chebyshev Type-I polynomial basis for physics-informed PDE solving. Its Cheby1KAN layer can be written as Φk,i(xi′)=∑n=0NCk,i,nTn(xi′)Φ_{k,i}(x'_i) = ∑_{n=0}^{N} C_{k,i,n} T_n(x'_i), where xi′=tanh(xi)x'_i = tanh(x_i) and Ck,i,nC_{k,i,n} are learnable coefficients. LEREN uses q(ℓ)=tanh(αℓ(Wℓh(ℓ−1)+bℓ))q^{(ℓ)} = tanh(α_ℓ(W_ℓ h^{(ℓ-1)} + b_ℓ)) and h(ℓ)=∑n∈SℓcℓnPn(q(ℓ))h^{(ℓ)} = ∑_{n ∈ S_ℓ} c_{ℓ n} P_n(q^{(ℓ)}). The two constructions are conceptually close. Both combine normalization, an orthogonal polynomial basis, learnable polynomial coefficients, and deep composition. A direct comparison with AC-PKAN would therefore be important for clarifying whether the empirical advantage comes specifically from Legendre polynomials or from the broader design pattern of learnable orthogonal-polynomial representations.

[1] AC-PKAN: Attention-Enhanced and Chebyshev Polynomial-Based Physics-Informed Kolmogorov-Arnold Networks, TMLR 2026.

Questions for authors

Can the authors establish a more formal connection between the polynomial composition argument and the actual LEREN architecture? Equation 11 shows that composing Legendre polynomials can generate higher-degree Legendre modes. However, the actual LEREN architecture in Eqs. 13–14 does not directly compose Legendre polynomials. Each layer also contains an affine transformation and a tanh mapping before the next Legendre expansion. Can the authors clarify whether the higher-mode generation argument in Eq. 11 still applies to the full architecture? A theoretical characterization of how affine mixing and tanh transformations affect the compositional mode-generation mechanism would make the proposed motivation more convincing.

Please clarify the value of cinitc_{\rm init} used in the Diffusion experiment. Section 3 derives the first-layer initialization c=128211.c=\frac{128}{211}. Appendix D.6 also states that all experimental results use the first-layer odd-coefficient initialization 128/211128/211. However, Table 6 reports the Diffusion result as Ours (cinit=0.2),\text{Ours }(c_{\rm init}=0.2), with an RL2 error of 3.199×10−33.199\times10^{-3}, which corresponds to the Diffusion result reported in the main table. Which initialization was actually used for the reported Diffusion result? If cinit=0.2c_{\rm init}=0.2 was used, please explain why the theoretically motivated initialization 128/211128/211 was not adopted for this benchmark. It would also be important to clarify whether 0.20.2 was selected through benchmark-specific tuning. This issue directly affects reproducibility and the claimed generality of the proposed initialization strategy.

Can the authors provide a frequency-scaling experiment? The current evidence for high-frequency approximation is mainly based on a fixed convection setting with c=80c=80 and a fixed Helmholtz setting with a1=a2=100a_1=a_2=100. It would be useful to evaluate the method over a range of frequencies, for example a∈{10,25,50,100,200},a\in\{10,25,50,100,200\}, and report how the errors of LEREN, Fourier-based baselines, and a standard tanh baseline change as the target frequency increases. If the paper aims to argue that LEREN alleviates high-frequency approximation difficulties, such a frequency-scaling experiment would provide substantially stronger evidence than evaluation at a single high-frequency operating point.

11 comments

We thank the reviewer for the detailed feedback and suggestions. We have strengthened the theoretical analysis and added experiments, and address the related weaknesses (W) and questions (Q) together below.

W1 and Q1. Connecting the actual architecture to a Legendre representation

To establish the connection between the actual layer and its Legendre representation, we have added Theorem 1 to the main text, with the full proof in Appendix N. Starting from the Legendre expansion of tanh⁡(wx+b)\tanh(wx+b), the theorem expresses the complete layer in the Legendre basis of the input coordinate:

=\sum_{k=0}^{\infty}\left(\sum_{n=0}^{N}c_nT_{kn}(w,b)\right)P_k(x).$$ The coefficients $T_{kn}(w,b)$ are constructed using the Legendre recurrence, and the expansion is unique and converges absolutely and uniformly on $[-1,1]$. This gives an explicit connection between the trainable layer coefficients and the resulting Legendre coefficients, enabling composition analysis with the affine transformation and $\tanh$ included. Building on this representation, the revised universal-approximation argument retains a nonconstant Legendre combination, without setting the higher-degree terms to zero and reducing the model to a standard $\tanh$ network. The construction also extends to multidimensional inputs through tensor-product Legendre bases. We thank the reviewer for this suggestion, which helped us strengthen the theoretical foundation of our work. > W2. Learning higher-degree modes through gradient-based optimization Theorem 1 shows that the Legendre spectral coefficients of each layer are controlled by the network parameters $w$ and expansion coefficients $c$. Gradient updates to $w$ and $c$ therefore adjust these spectral coefficients, including those of higher-degree modes, allowing these modes to be learned from the PDE training loss. The component ablation in Appendix B, Table 5 supports the effectiveness of this joint optimization. We compare the $\tanh$ baseline, fixed Legendre coefficients, and learnable Legendre coefficients using the same four-layer backbone, trainable activation scales, and shared readout. All variants use Adam, seed $42$, and the same training budget within each task. Their relative $L_2$ errors are shown below. | Configuration | Convection | Helmholtz | | :--- | :--- | :--- | | $\tanh$ with the same backbone | $0.912$ | $0.247$ | | Fixed Legendre coefficients | $2.09\times10^{-4}$ | $2.35\times10^{-2}$ | | Full LEREN | **$9.33\times10^{-5}$** | **$7.10\times10^{-3}$** | With $c$ fixed, learning the network parameters already gives much lower errors than the $\tanh$ baseline. Jointly learning $w$ and $c$ further reduces the errors on both equations, demonstrating the benefit of learning the Legendre representation through gradient updates. > W3. Direct evidence of higher-degree modes in hidden representations To demonstrate that layer composition generates higher-degree modes, we projected the learned hidden features onto a Legendre basis layer by layer. On convection, we trained one- to five-layer models with width $256$ and maximum activation degree $9$, then projected their hidden features along time onto $P_n(2t-1)$ for $n=0,\ldots,128$. The projections show stronger high-degree components in deeper layers (Appendix J, Figure 6). In the four-layer model, the energy-weighted mean degrees increase from $1.64$ to $3.08$, $7.16$, and $12.29$, and the later layers contain more energy around degree $64$.

W4. Choice and generality of the layerwise degree sets

We chose 0,1,8,9\\{0,1,8,9\\} based on a simple denoising idea: retain the components needed to fit the signal while reducing unnecessary terms in each later layer. We keep two low-degree terms and two high-degree terms:

  • P0,P1P_0,P_1 provide the constant and linear terms.
  • P8,P9P_8,P_9 provide higher-degree changes. With maximum degree 99, these are the highest even and odd degrees, so we retain both types.

Further increases in depth and degree can make training harder by introducing more components that must be suppressed. For example, f(x)=sin⁡x+sin⁡(10x)f(x)=\sin x+\sin(10x) only needs two frequencies. If a network generates many additional components, it must learn to suppress them or cancel their contributions; unwanted components that remain cause fitting error. Figure 4 shows that four layers and maximum degree 99 achieve the lowest error, while further increases raise the error.

We retain the full P0P_0–P9P_9 set in the first layer because it acts directly on the input coordinates and provides a richer starting representation. Later layers combine these learned features using P0,P1,P8,P9P_0,P_1,P_8,P_9. All selected coefficients remain trainable.

The added ablation tests this choice on both convection and Helmholtz (Appendix F, Table 9). The sparse variant uses P0,P1,P8,P9P_0,P_1,P_8,P_9 in every layer, and the full variant uses P0P_0–P9P_9 in every layer. Relative L2L_2 errors are reported as Mean ±\pm Std.

ConfigurationConvectionHelmholtz
Default(1.421±0.4318)×10−4(1.421\pm0.4318)\times10^{-4}(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}
Sparse in all layers(2.253±1.388)×10−4(2.253\pm1.388)\times10^{-4}(1.171±1.155)×10−2(1.171\pm1.155)\times10^{-2}
Full in all layers(1.625±0.7091)×10−4(1.625\pm0.7091)\times10^{-4}(4.613±3.179)×10−2(4.613\pm3.179)\times10^{-2}

Using all ten terms in every layer does not improve either benchmark. Using only four terms in every layer improves Helmholtz but worsens convection. We therefore retain the full first layer and sparse later layers as the shared default.

W5. A more challenging coupled-flow benchmark

To evaluate LEREN on more complex nonlinear flow with steep fronts, we added a one-dimensional viscous compressible Navier–Stokes shock-tube experiment (Appendix I, Table 13). The problem jointly predicts density, velocity, and pressure, with viscosity μ=10−3\mu=10^{-3}. We compare four methods over three seeds under approximately matched training-time budgets. LEREN achieves the lowest mean error and standard deviation, with a mean error 28.3% below JAX-PI.

MethodRelative L2L_2 error (Mean ±\pm Std.)
DGM0.2054±0.017130.2054\pm0.01713
JAX-PI0.1720±0.011800.1720\pm0.01180
IFeF-PINN0.2176±0.012930.2176\pm0.01293
LEREN0.1233±0.0072080.1233\pm0.007208

W6. Comparison with AC-PKAN

We have added a direct comparison with the complete AC-PKAN method, retaining its Chebyshev basis and attention mechanism. Appendix H, Table 12 compares AC-PKAN, a B-spline KINN baseline, and LEREN on three benchmarks under matched training-time budgets, with each method retaining its own training strategy.

LEREN achieves lower mean errors than both KAN baselines on all three benchmarks. The relative L2L_2 errors are reported as Mean ±\pm Std.

BenchmarkAC-PKANKINNLEREN
Convection8.027±3.5028.027\pm3.5022.087±0.021812.087\pm0.02181(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
Burgers0.4812±0.13780.4812\pm0.13780.4549±0.0056730.4549\pm0.005673(4.668±0.06800)×10−5(4.668\pm0.06800)\times10^{-5}
Helmholtz15.43±3.41215.43\pm3.41256.99±0.364756.99\pm0.3647(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}

LEREN learns Legendre representations within an MLP architecture. AC-PKAN learns polynomial coefficients on each edge, whereas LEREN first applies an affine transformation and then a Legendre combination with coefficients shared within each layer.

Q2. Initialization used in the diffusion experiment

c=0.2c=0.2 is an old setting, which has been replaced by c=128/211c=128/211 in our current default.

The current initialization follows the unit-slope condition ϕ1′(0)=1\phi_1'(0)=1. The added multi-seed evaluation in Appendix D uses 128/211128/211 across all five equations, including diffusion.

The five-benchmark comparison shows more stable results and better overall accuracy with c=128/211c=128/211. As reported in Appendix E, Table 8, it yields a lower error standard deviation on all five benchmarks and a lower mean error on four. Relative L2L_2 errors are reported as Mean ±\pm Std.

Benchmarkc=0.2c=0.2c=128/211c=128/211
Convection(9.158±11.73)×10−4(9.158\pm11.73)\times10^{-4}(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
Burgers(5.169±0.1807)×10−5(5.169\pm0.1807)\times10^{-5}(4.668±0.06800)×10−5(4.668\pm0.06800)\times10^{-5}
Helmholtz0.6465±0.60240.6465\pm0.6024(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}
Heat-5D(1.274±1.825)×10−3(1.274\pm1.825)\times10^{-3}(1.396±1.003)×10−3(1.396\pm1.003)\times10^{-3}
Nonlinear diffusion(4.719±0.8117)×10−3(4.719\pm0.8117)\times10^{-3}(4.230±0.4041)×10−3(4.230\pm0.4041)\times10^{-3}

Q3. Frequency-scaling experiments

To evaluate performance as the target frequency increases, we added the requested Helmholtz scan over a∈10,25,50,100,200a\in\\{10,25,50,100,200\\} and a convection scan over c∈40,80,100,200c\in\\{40,80,100,200\\}. Appendix G, Tables 10–11 compare LEREN, a standard tanh⁡\tanh MLP, and a fixed multiscale Fourier-feature MLP, all with four hidden layers of width 256256. The Fourier projections remain fixed throughout each scan.

LEREN achieves the lowest mean error in most tested settings, demonstrating its ability to handle both high- and low-frequency solutions. The relative L2L_2 errors are reported as Mean ±\pm Std.

Convection:

cctanh⁡\tanh MLPFourier-feature MLPLEREN
40(5.133±3.023)×10−3(5.133\pm3.023)\times10^{-3}(1.611±0.08710)×10−3(1.611\pm0.08710)\times10^{-3}(1.063±0.2168)×10−4(1.063\pm0.2168)\times10^{-4}
800.5878±0.12740.5878\pm0.1274(6.342±0.2864)×10−3(6.342\pm0.2864)\times10^{-3}(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
1001.047±0.085691.047\pm0.08569(2.376±0.3756)×10−2(2.376\pm0.3756)\times10^{-2}(4.575±7.252)×10−2(4.575\pm7.252)\times10^{-2}
2001.005±0.046491.005\pm0.046490.9376±0.0044710.9376\pm0.0044711.085±0.11251.085\pm0.1125

Helmholtz:

aatanh⁡\tanh MLPFourier-feature MLPLEREN
10(4.738±0.7531)×10−5(4.738\pm0.7531)\times10^{-5}1.006±0.0028931.006\pm0.002893(6.499±7.173)×10−5(6.499\pm7.173)\times10^{-5}
25(4.113±1.370)×10−4(4.113\pm1.370)\times10^{-4}0.9752±0.0074520.9752\pm0.007452(1.174±1.099)×10−4(1.174\pm1.099)\times10^{-4}
50(7.430±2.602)×10−3(7.430\pm2.602)\times10^{-3}0.8843±0.026840.8843\pm0.02684(7.149±4.326)×10−4(7.149\pm4.326)\times10^{-4}
1005.141±3.5775.141\pm3.5770.4808±0.047080.4808\pm0.04708(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}
2006.633±5.1586.633\pm5.1580.5303±0.013430.5303\pm0.013433.629±1.7003.629\pm1.700
Reviewer_xCHqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a variant in physics-informed neural networks (PINNs) that addresses the challenge of accurately approximating the high-frequency PDEs. The proposed method, Legendre Representation Networks (LERENs), extends the interpretation of sinusoidal networks as implicit spectral representations to Legendre representations through nested polynomial composition.The authors challenge the need for spectral priors in representation design and consider the commonly used Fourier representations in PINNs and SIREN can be sensitive to frequency selection, embedding dimension, and initialization. In contrast, LERENs has the contribution of approximating high-frequency PDE solutions without Fourier embeddings or sinusoidal activations, with the spectral domain of the network controlled through the selected Legendre degrees. Experiments on several challenging benchmarks demonstrate the advantages of Legendre representations over Fourier-based PINNs and other advanced PINN variants.

Critical strengths
  1. The proposed LERENs can approximate high-frequency PDE solutions without Fourier embeddings or sinusoidal activations.

  2. Introducing the composition of Legendre polynomials into the network with activation functions from the observation to SIREN is concise and effective. It introduces few learnable parameters and does not introduce too much computational cost.

  3. The ablation study in the paper is concrete. It compares a tanh network, LERENs with frozen coefficients, and full LERENs, which clearly separates the contribution of the Legendre expansion from that of coefficient learning, and provides convincing evidence that the expansion is the main component contributing to the improvement.

Critical weaknesses
  1. The implicit Legendre representation is not actually derived. The paper attempts to obtain an implicit Legendre representation similar to the TUNER derivation. However, TUNER's representation is mathematically derived from the composition sin⁡(sin⁡(⋅))\sin(\sin(\cdot)). The LEREN layer now has the composition form Pn(tanh⁡(⋅))P_n(\tanh(\cdot)). It is not straightforward how this form can be further expanded into an implicit spectral representation. The paper's derivation covers Eqs. (11)–(15) and stops at the definition of the network output, without showing that this output actually admits a Legendre expansion with respect to the input coordinates, similarly to TUNER.

  2. The universal approximation result is too plain. In Section 3.2, the proof sets α1=1\alpha_1 = 1, c11=1c_{11} = 1, and all other expansion coefficients to zero, which means the network then degenerates into the MLP. This proof does not involve any theoretical analysis of how the LEREN structure affects the approximation capability of the network.

  3. In the abstract and introduction, the authors *challenge the need for spectral priors in representation design and argue that Fourier-based PINNs are sensitive to frequency selection. However, choosing the Legendre polynomials PnP_n in LERENs is also a prior selection of the frequency for the network. This contradicts the claims in the abstract and introduction.

  4. The experimental setup is not clearly stated. Appendix D.6 states that LERENs are trained together with auxiliary techniques such as residual point resampling, causal weighting, and gradient-norm-based loss balancing, but it is not stated whether the baselines use the same settings. In addition, no mean errors or standard deviations are reported for the results, which means all experiments may have just one run.

Questions for authors
  1. Why set the default 4-layer architecture? Moreover, why does combining deeper networks with higher polynomial degrees increase the accuracy in Figure 4? This seems contrary to the motivation that composition generates higher-degree modes and should therefore benefit high-frequency approximation.

  2. Since LERENs are motivated by extending the SIREN/TUNER, the authors should include SIREN-based baseline into account but not just Fourier-based.

  3. Following Weaknesses 1 and 2, could the authors provide a more meaningful analysis, e.g., (i) a derivation showing the LEREN output constitutes an implicit Legendre representation, and (ii) an approximation result that reflects the role of the Legendre expansions and their composition?

  4. Why does the Diffusion problem in Table 7 use a different seed (113) from the other problems (42)? Regarding Weakness 4, could the authors describe in more detail the experimental settings of LERENs and of each baseline (if the baselines use the same component with LERENs, such as causal weighting...), and report the mean and standard deviation over multiple random seeds?

2 comments

We thank the reviewer for the detailed feedback and suggestions. We have strengthened the theoretical analysis and added experiments, and address the related weaknesses (W) and questions (Q) together below.

W1–W2 and Q3. Deriving the implicit Legendre representation and its approximation capability

To derive the implicit Legendre representation with tanh⁡\tanh included, we have added Theorem 1 to the main text and its full proof to Appendix N. Starting from the Legendre expansion of tanh⁡(wx+b)\tanh(wx+b), the theorem expresses the complete layer in the Legendre basis of the input coordinate:

=\sum_{k=0}^{\infty}\left(\sum_{n=0}^{N}c_nT_{kn}(w,b)\right)P_k(x).$$ The coefficients $T_{kn}(w,b)$ are constructed using the Legendre recurrence, and the expansion is unique and converges absolutely and uniformly on $[-1,1]$. This gives an explicit connection between the trainable layer coefficients and the resulting Legendre coefficients, making it possible to analyze composition in a common Legendre basis. The same construction extends to multidimensional inputs through tensor-product Legendre bases. The revised universal-approximation argument retains a nonconstant Legendre combination, without setting the higher-degree terms to zero and reducing the model to a standard $\tanh$ network. We thank the reviewer for the suggestion, which helped us strengthen the theoretical foundation of our work. To examine higher-degree components in the trained network, we also added layerwise spectral projections in Appendix J, Figure 6. We trained one- to five-layer models on convection and projected their hidden features along time onto $P_n(2t-1)$ for $n=0,\ldots,128$. In the four-layer model, the energy-weighted mean degrees increase from $1.64$ to $3.08$, $7.16$, and $12.29$ across layers. These results provide direct experimental evidence that the learned representations develop stronger high-degree components through successive layers. > W3. On spectral priors By “spectral priors,” we refer to two types of prior information: 1. **Choosing a representation based on prior knowledge of the solution's form.** For example, one may choose a Fourier method for convection because its solution contains trigonometric functions, while choosing another representation for a heat equation whose solution does not. Such choices use prior knowledge of the target solution's structure. 2. **Choosing the initialization based on prior knowledge of the solution's frequencies.** For example, knowing the approximate target frequencies allows one to initialize the Fourier frequency distribution near them. Even when the frequencies are learnable, this initialization still uses spectral information about the target. We also observed the influence of frequency scale with SIREN: our configuration uses a first-layer frequency factor of $30$, but struggles to fit the low-frequency Helmholtz problem with $a=4$. LEREN aims to reduce dependence on both types of prior information. We use the same Legendre representation across different equations, including convection and Heat-5D, without switching representations according to whether the solution contains trigonometric functions. The coordinate projections and Legendre coefficients are learned during training, without prescribing the target solution's frequencies.

Q1. Why four layers, and why can further increases in depth and degree reduce accuracy?

We chose four layers because composition already provides rich high-degree components at this depth, and the combination of four layers and maximum degree 99 achieved the lowest error in Figure 4. Our aim is to provide sufficient representational capacity while maintaining effective training.

Composition can increase the attainable polynomial degree rapidly. As an intuitive example of pure polynomial composition, composing four degree-99 polynomials gives maximum degree 94=65619^4=6561. Increasing the depth to five gives 95=590499^5=59049, while keeping four layers and increasing the degree to 1111 gives 114=1464111^4=14641. Thus, modest layerwise degrees can already generate very high degrees through composition. The actual network containing tanh⁡\tanh admits the Legendre expansion established by Theorem 1.

The target solution may not require so many high-degree components. For example, if a target can already be approximated accurately using around 10001000 Legendre modes, introducing additional high-degree components does not automatically improve accuracy. The network must also learn to suppress unnecessary components or cancel them through combinations of features. Any unwanted components that remain contribute to the approximation error.

The experiments support this trade-off. Figure 4 shows that increasing depth and degree initially improves accuracy, with the best result obtained at four layers and maximum degree 99; further increases raise the error. The ablation in Appendix F also shows that retaining all ten terms in every layer does not improve either convection or Helmholtz. We therefore retain four layers, with P0P_0–P9P_9 in the first layer and P0,P1,P8,P9P_0,P_1,P_8,P_9 in the subsequent layers, as our default configuration.

Q2. Scope of the SIREN comparisons and the omission of TUNER

We did not extend the SIREN comparisons to every equation primarily because of its sensitivity to frequency initialization. For example, our SIREN configuration uses a first-layer frequency factor of 3030 to introduce rich high-frequency components, but struggles to fit the low-frequency Helmholtz problem with a=4a=4. This illustrates the difficulty of applying the same frequency setting across different PDEs without problem-specific adjustment. We therefore did not directly extend this configuration to all benchmarks.

We did not include TUNER because its frequency sampling scheme, designed for image reconstruction, uses a frequency range that is too large for our PINN settings. Applying it to the PDE tasks studied here would require adjusting the sampling range and strategy. We therefore draw on TUNER’s theoretical analysis of frequency generation through layer composition, while not directly adopting its original training scheme as a PINN baseline.

W4 and Q4. Random seeds and auxiliary training settings

Diffusion used seed 113113 because the original manuscript reported the best-performing run among the seeds we had tested. To assess the effect of random seeds, we have added experiments with seeds 4242, 113113, and 111111 on all five benchmarks using the current default LEREN configuration. The architecture and training settings are held fixed within each benchmark. Appendix D, Table 7 reports the following relative L2L_2 errors as Mean ±\pm Std.:

BenchmarkLEREN (Mean ±\pm Std.)
Convection(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
Burgers(4.668±0.06800)×10−5(4.668\pm0.06800)\times10^{-5}
Helmholtz(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}
Heat-5D(1.396±1.003)×10−3(1.396\pm1.003)\times10^{-3}
Nonlinear diffusion(4.230±0.4041)×10−3(4.230\pm0.4041)\times10^{-3}

The added results show relatively stable performance on diffusion, with a mean error of 4.230×10−34.230\times10^{-3} and a standard deviation of 4.041×10−44.041\times10^{-4}.

JAX-PI, PirateNet, and LEREN share an auxiliary training framework, using components such as causal weighting, gradient-norm loss balancing, and random weight factorization where applicable. IFeF-PINN and ProPINN retain their own optimization procedures without these additional components, allowing us to evaluate their complete methods. We have added the full training settings for each method to Appendix M, including the architecture, collocation and resampling strategy, optimizer, loss weights, and auxiliary training components.

Reviewer_xykqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LERENs (Legendre Representation Networks), a way to solve PDEs with high-frequency solutions using physics-informed neural networks (PINNs) without Fourier features or sinusoidal activations.

Neural networks tend to learn smooth, low-frequency parts of a function first. This spectral biases make PINNs struggle with rapidly oscillating PDE solutions, and current fixes rely on sines and cosines by Fourier feature embeddings of the inputs, for example. The authors argue that in SIREN, stacking sin layers produces new frequencies that are integer combinations of the input frequencies. Legendre polynomials behave similarly; composing the layers gives a polynomial of degree and generates higher-degree modes.

In the experiments, the authors show that LEREN performs great under five different benchmarks, also reducing the errors significantly using the same backbone throughout as compared to Plain tanh.

Critical strengths

(1) The authors propose an idea that is simple and costs no extra parameters. The LEREN solution is quite elegant and easy to follow.

(2) The authors have provided some theoretical understandings of why LEREN should work based on observations about Legendre polynomials and universal function approximations.

(3) The experiments show that LEREN performs well. The convection and Burgers gains over strong baselines are large.

(4) The authors have provided some ablations on from purely the tanh functions to show that gains from LEREN is real.

Critical weaknesses

(1) The universal function approximation theory seems trivial to me. The proof sets every coefficient except c1c_1 to zero, which turns the network into an ordinary tanh network. It says nothing about why Legendre expansions help (only shows tanh helps).

(2) The argument about the Legendre degree-growth with stacking layers is also a bit heuristic. Because each layer applies tanh(affine function()) before the polynomials the networks never composes PmP_m and PnP_n directly.

(3) The experiment results are single runs. The authors should report the error bars and confidence intervals across multiple runs. On some datasets Heat-5D and diffusions, the margins are so small and improvements may just due to randomness within the seed noise.

(4) Some initialization settings are tuned per benchmark. The diffusion run uses cinitc_\text{init} = 0.2, and the appendix mentions benchmark-specific variants. This weakens the claims about the initialization is principled.

(5) Lack of related works. There have been many works closely related to this line of research: Chebyshev or Legendre KANs and to other learnable-activation networks. The authors should also mention other baselines (e.g., transformer based architectures, or adaptive learning).

Questions for authors

(1) Can the authors show that composition actually generate higher-degree modes? For example, can the authors project learned hidden features onto Legendre or Fourier bases layer by layer and check whether the dominant degree or frequency rises with depth?

(2) Does the author consider replacing tanh() with other activation functions?

(3) Why the degree set {0,1,8,9}\{0, 1, 8, 9\}?

(4) How does LEREN perform with harder tasks? For example a=200a = 200 or 400400 in Helmholtz and convection to c=100c = 100 or more. Where does LEREN break compared with Fourier methods?

(5) How does it compare with KANs, different training dynamics?

(5) Have the authors consider to try shocks with lower Burgers viscosity, Allen–Cahn, Kuramoto–Sivashinsky, or Navier–Stokes, where non-smooth or multiscale solutions exist, and polynomials may oscillate near discontinuities?

5 comments

We thank the reviewer for the detailed feedback and concrete experimental suggestions. We address the weaknesses (W) and questions (Q) below.

W1–W2. Legendre representations, composition, and universal approximation

To address both concerns, we have added Theorem 1 to the main text, with the full proof in Appendix N. The theorem expresses the layer containing tanh⁡\tanh as an expansion in Legendre polynomials, enabling us to analyze network composition in a common Legendre basis.

Building on this representation, the revised universal-approximation argument retains a nonconstant Legendre combination, without setting the higher-degree terms to zero and reducing the model to a standard tanh⁡\tanh network. We thank the reviewer for this suggestion, which helped us strengthen the theoretical foundation of our work.

W3. Variability across random seeds

To address the concern that single-run results may reflect seed variability, we added a three-seed evaluation on all five benchmarks. Appendix D, Table 7 reports the mean and sample standard deviation of the relative L2L_2 errors.

BenchmarkMeanStd.
Convection1.816×10−41.816\times10^{-4}8.931×10−58.931\times10^{-5}
Burgers4.668×10−54.668\times10^{-5}6.800×10−76.800\times10^{-7}
Helmholtz2.014×10−22.014\times10^{-2}1.842×10−21.842\times10^{-2}
Heat-5D1.396×10−31.396\times10^{-3}1.003×10−31.003\times10^{-3}
Nonlinear diffusion4.230×10−34.230\times10^{-3}4.041×10−44.041\times10^{-4}

W4. Initialization and benchmark-specific settings

The c=0.2c=0.2 setting was used in an early version and has been replaced by c=128/211c=128/211 in our current default.

The five-benchmark comparison shows more stable results and better overall accuracy with c=128/211c=128/211. As reported in Appendix E, Table 8, it yields a lower error standard deviation on all five benchmarks and a lower mean error on four of the five benchmarks.

Relative L2L_2 error (Mean ±\pm Std.).

Benchmarkc=0.2c=0.2c=128/211c=128/211
Convection(9.158±11.73)×10−4(9.158\pm11.73)\times10^{-4}(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
Burgers(5.169±0.1807)×10−5(5.169\pm0.1807)\times10^{-5}(4.668±0.06800)×10−5(4.668\pm0.06800)\times10^{-5}
Helmholtz(6.465±6.024)×10−1(6.465\pm6.024)\times10^{-1}(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}
Heat-5D(1.274±1.825)×10−3(1.274\pm1.825)\times10^{-3}(1.396±1.003)×10−3(1.396\pm1.003)\times10^{-3}
Nonlinear diffusion(4.719±0.8117)×10−3(4.719\pm0.8117)\times10^{-3}(4.230±0.4041)×10−3(4.230\pm0.4041)\times10^{-3}

W5 and Q5. Related work, KANs, and training dynamics

To address the missing related work and KAN comparisons, we added KAN baselines and will expand the related-work discussion. The new experiments compare complete AC-PKAN (H. Zhang et al., 2026) and a B-spline KINN baseline (Wang et al., 2025) on three benchmarks (Appendix H, Table 12).

LEREN achieves lower mean errors than both KAN baselines under matched training-time budgets. The final relative L2L_2 errors (Mean ±\pm Std.) are shown below.

BenchmarkAC-PKANKINNLEREN
Convection8.027±3.5028.027\pm3.5022.087±0.021812.087\pm0.02181(1.816±0.8931)×10−4(1.816\pm0.8931)\times10^{-4}
Burgers(4.812±1.378)×10−1(4.812\pm1.378)\times10^{-1}(4.549±0.05673)×10−1(4.549\pm0.05673)\times10^{-1}(4.668±0.06800)×10−5(4.668\pm0.06800)\times10^{-5}
Helmholtz15.43±3.41215.43\pm3.41256.99±0.364756.99\pm0.3647(2.014±1.842)×10−2(2.014\pm1.842)\times10^{-2}

Regarding training dynamics, KANs can be more difficult to optimize than MLPs and often require longer training, even with fewer parameters. For example, for the same number of training steps, LEREN takes approximately 1 hour, whereas KAN requires approximately 18 hours. This is one reason we explore Legendre representations within an MLP architecture.

Q1. Direct evidence of higher-degree modes in hidden representations

To demonstrate that layer composition generates higher-degree modes, we projected the learned hidden features onto a Legendre basis layer by layer. On the convection equation, we trained one- to five-layer models with width 256 and maximum activation degree 9, then projected the hidden features along time onto Pn(2t−1)P_n(2t-1) for n=0,…,128n=0,\ldots,128.

The projections show stronger high-degree components in the deeper hidden representations (Appendix J, Figure 6). In the four-layer model, the energy-weighted mean degrees increase from 1.64 to 3.08, 7.16, and 12.29 across layers, and the later layers contain more energy around degree 64.

Q2. Alternatives to the tanh input map

To evaluate alternatives to tanh⁡\tanh, we compared four replacement input mappings. Appendix C, Table 6 compares tanh⁡\tanh with the identity map, s/1+s2s/\sqrt{1+s^2}, LayerNorm, and RMSNorm. The comparison supports retaining tanh⁡\tanh, which achieves the lowest mean errors on both convection and Helmholtz.

ddsPn(tanh⁡s)=n(Pn−1(q)−qPn(q)),q=tanh⁡s.\frac{d}{ds}P_n(\tanh s)=n\bigl(P_{n-1}(q)-qP_n(q)\bigr),\qquad q=\tanh s.

Q3. Why use the degree set 0,1,8,9\\{0,1,8,9\\}?

We chose 0,1,8,9\\{0,1,8,9\\} based on a simple denoising idea. We want to keep the components needed to fit the signal without introducing too many unnecessary ones in every layer.

The motivation is easy to see with f(x)=sin⁡x+sin⁡(10x)f(x)=\sin x+\sin(10x). This signal only needs frequencies 1 and 10. If a network generates thousands of frequency components, it must also learn to suppress the extra ones or cancel their contributions through combinations of features. Any unwanted components left over cause fitting error. TUNER also uses frequency selection and filtering to control unnecessary components.

We retain two low-degree terms and two high-degree terms, reducing each later-layer expansion from ten terms to four:

  • P0,P1P_0,P_1 provide the constant and linear terms.
  • P8,P9P_8,P_9 provide higher-degree changes. With the maximum degree set to 9, these are the highest even and odd degrees, so we keep both types.

A small set of layerwise degrees can still generate much higher-degree terms through composition. For example, consider pure polynomial composition without tanh⁡\tanh: P1(P9(x))P_1(P_9(x)) stays at degree 9, P8(P9(x))P_8(P_9(x)) reaches degree 72, and P9(P9(x))P_9(P_9(x)) reaches degree 81. This illustrates why we can keep only a few terms in each layer and still generate higher-degree structure by stacking layers. Q1 shows the growth of high-degree components in the actual network.

We keep the full P0P_0–P9P_9 set in the first layer because it acts directly on the input coordinates. This gives the network a richer starting representation. Later layers combine these learned features using P0,P1,P8,P9P_0,P_1,P_8,P_9, without adding every intermediate degree explicitly.

The added ablation supports this choice. In Appendix F, Table 9, using all ten terms in every layer did not improve either benchmark. Using only four terms in every layer improved Helmholtz but worsened convection. We therefore keep all ten terms in the first layer and four in the later layers.

Q4. Higher frequencies and comparison with Fourier features

To evaluate harder tasks and identify where LEREN loses accuracy relative to Fourier features, we extended the frequency scan for convection and Helmholtz. Appendix G (Tables 10–11) compares LEREN, a tanh⁡\tanh MLP, and a fixed multiscale Fourier-feature MLP over c∈40,80,100,200c\in\\{40,80,100,200\\} for convection and a∈10,25,50,100,200a\in\\{10,25,50,100,200\\} for Helmholtz. LEREN achieves the lowest mean error in most tested settings.

Beyond the high-frequency comparisons above, LEREN offers three practical advantages over the tested Fourier-based representations.

  1. Reduced dependence on auxiliary modules. LEREN achieves accurate solutions without periodic embeddings or a ModifiedMLP backbone. Main-text Table 1 shows that these components substantially affect the Fourier-based baselines: on convection, the full JAX-PI configuration achieves an error of 6.88×10−46.88\times10^{-4}, which increases to 7.38×10−17.38\times10^{-1} without periodic embeddings and 1.02×10−21.02\times10^{-2} without ModifiedMLP.

    The tested SIREN configuration also remains less accurate despite using periodic embeddings. Its convection error is 1.88×10−11.88\times10^{-1}. These comparisons show that introducing sinusoidal features alone does not ensure accurate PDE solutions, whereas LEREN achieves high accuracy without these additional architectural components.

  2. No prescribed target-frequency initialization. Fourier representations can remain sensitive to their initial frequency distribution even when the frequencies are trainable. In our main comparisons, we initialized Fourier-based methods with frequency ranges close to the target spectrum; this uses prior spectral knowledge. If the initial distribution is far from the relevant range, recovering the target frequencies can become difficult.

    SIREN illustrates the importance of matching the frequency scale to the task. The SIREN configuration we use has a first-layer frequency factor of 30 to introduce rich high-frequency variation, but it is difficult to fit the low-frequency Helmholtz case with a=4a=4. TUNER's frequency-control scheme also cannot be transferred directly to our PINN setting and requires additional adaptation.

    LEREN learns its coordinate projections and Legendre coefficients without prescribing the solution frequencies. This allows it to mitigate spectral bias without target-specific Fourier initialization. The frequency scan in Appendix G further shows that LEREN remains accurate at lower Helmholtz frequencies where the same fixed Fourier mapping performs poorly.

  3. Applicability beyond oscillatory solution profiles. Not all PDE solutions have a periodic or predominantly sinusoidal structure, so a sinusoidal basis is not always the most natural choice. Basis selection should account for the domain, boundary conditions, and solution structure; Legendre polynomials provide a natural alternative for bounded, nonperiodic problems.

    Our Heat-5D and nonlinear-diffusion results demonstrate that LEREN also works beyond explicitly sinusoidal solutions. Its value therefore extends beyond reproducing high-frequency oscillations: it provides a way to learn polynomial spectral representations within an MLP and motivates further exploration of Legendre and other orthogonal bases for PDE learning.

Q6. Coupled flow and steep solution fronts

To assess LEREN on PDEs with steep solution fronts, we added a viscous compressible Navier–Stokes shock-tube experiment. LEREN achieves the lowest mean error and error standard deviation, with a mean error 28.3% below JAX-PI, the strongest baseline in this comparison (Appendix I, Table 13).

MethodRelative L2L_2 error (Mean ±\pm Std.)
DGM(2.054±0.1713)×10−1(2.054\pm0.1713)\times10^{-1}
JAX-PI(1.720±0.1180)×10−1(1.720\pm0.1180)\times10^{-1}
IFeF-PINN(2.176±0.1293)×10−1(2.176\pm0.1293)\times10^{-1}
LEREN(1.233±0.07208)×10−1(1.233\pm0.07208)\times10^{-1}

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.