acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Multi-Frequency Representations for Physics-Informed Neural Networks with Structured Variable Separation

Abstract

Physics-informed neural networks (PINNs) often struggle with multiscale, high-dimensional partial differential equations (PDEs) because of spectral bias and inefficient representations of interactions among variables. We propose adaptive multi-frequency representations for PINNs with structured variable separation (MFAR-SVS). Each neuron combines multiple sinusoidal branches with jointly learnable frequencies and weights, allowing its spectral representation to adapt to the target solution's frequency content rather than relying on fixed or manually selected frequencies. To exploit structured dependencies in high-dimensional solutions, we introduce a bipartite low-rank factorization that preserves unrestricted interactions within each variable group while imposing low-rank coupling only across groups. This provides an intermediate representation between fully coupled networks and dimension-wise tensor decompositions. We derive a closed-form neural tangent kernel (NTK) for the resulting architecture and characterize its infinite-width and infinite-rank limits. Theoretical analysis shows that adaptive frequency learning introduces a positive-semidefinite correction to the fixed-frequency kernel, implying non-decreasing ordered eigenvalues, including the minimum eigenvalue. Controlled experiments further show that adaptive frequencies progressively capture higher-frequency components and yield larger minimum NTK eigenvalues. Across four PDE benchmarks, MFAR-SVS achieves consistent accuracy and convergence gains, with the largest improvements on multi-scale and strongly coupled problems.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics-informed neural networks, Spectral bias, Adaptive Multi-Frequency Representation, Structured Variable Separation, Neural Tangent Kernel

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wSeqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MFAR-SVS, a representation framework for physics-informed neural networks that combines adaptive multi-frequency sinusoidal activations with grouped low-rank variable separation. MFAR uses a weighted sum of sinusoidal branches with trainable frequency scales and mixing coefficients. SVS partitions the input variables into two predefined groups and combines the outputs of two subnetworks through a low-rank sum of products, allowing variables within each group, such as spatial coordinates, to be modeled jointly.

The paper presents a universal approximation result and an NTK decomposition for the grouped representation. It also analyzes the positive-semidefinite contribution of additional trainable parameters to the NTK under a comparison at the same parameter values. Experiments compare activation choices and factorization architectures on transport and Helmholtz problems, with additional Klein-Gordon and five-dimensional Poisson results in the appendix. The reported results suggest benefits from adaptive frequencies and spatial grouping on several transport problems, while some dimension-wise factorizations achieve lower errors on the separable Helmholtz benchmark.

Critical strengths

1. Clear and relevant motivation.

The paper addresses frequency representation and cross-variable coupling within a single framework. The two components, adaptive sinusoidal activations and grouped variable separation, correspond naturally to these challenges, making the motivation and overall architectural design easy to follow.

2. Useful empirical comparisons across architectures and problem settings.

The experiments consider several PDE families and compare fixed and adaptive frequency variants across multiple factorization architectures, ranks, and collocation densities. The reported improvements across different decomposition families suggest that adaptive multi-frequency representations can be useful across different factorization architectures, with particularly promising results on the transport problems. The paper also reports cases in which dimension-wise factorizations outperform SVS, particularly on the separable Helmholtz benchmark, helping clarify the scope and limitations of structured variable separation.

3. Supplementary code and data facilitate inspection of Figure 3.

The supplement provides plotting and training/model code, saved error fields, and per-seed metrics for the decomposed models in Figure 3. These materials provide a useful basis for inspecting the supplied implementations, checking how Figure 3 was generated, and examining the associated experimental settings.

Critical weaknesses

1. The methodological novelty is insufficiently established.

Jagtap et al., "Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions" (arXiv:2105.09513), already consider activations of the form ∑kαkϕk(ωkz)\sum_k \alpha_k\phi_k(\omega_k z) with trainable coefficients and scales. Choosing sinusoidal basis functions, αk=ck/K\alpha_k=c_k/K, and ωk=nk\omega_k=n_k recovers the MFAR activation form, including the layer-wise parameter sharing used in the supplementary implementation. This limits the novelty of the activation itself. Combining this activation family with grouped low-rank representations is a reasonable design, and the component comparisons provide evidence of its utility. However, the paper should more clearly distinguish this architectural combination from a new activation method and identify which findings constitute its main additional methodological contribution.

2. The optimization interpretation is not fully justified, and the infinite-width argument contains a gap.

The finite-width NTK decomposition and the eigenvalue monotonicity in Proposition 4 are valid under their stated assumptions. However, the PSD increment is a general consequence of adding trainable parameter directions at a common parameter state. It does not, by itself, provide an explanation specific to frequency adaptation or establish an ordering between kernels along independently optimized training trajectories.

The connection to PINN optimization also requires more careful qualification. Even under gradient flow for the squared PDE/initial/boundary residual objective, the relevant kernel is formed from the Jacobian of the stacked, appropriately weighted residuals. The reported output-NTK spectra alone therefore do not establish improved convergence for this residual objective. The authors should clarify the scope of the optimization interpretation and distinguish the kernel comparison at a common parameter state from the training behavior it is intended to explain.

There is also a specific issue in Appendix A.6, Step 3, lines 931-934. Under Eq. (31), the frequency scales and branch weights are shared across hidden units. For fixed KK, their gradients are W−1/2W^{-1/2}-scaled sums whose outer products can retain non-vanishing random fluctuations as width increases. The asserted convergence to deterministic expectations through a law-of-large-numbers argument is therefore not justified as written. This step requires correction, although it does not invalidate the finite-width PSD identity.

3. Several experimental specifications and reported quantities require reconciliation.

In the Figure 3 supplementary package, the nondecomposed adaptive panels labeled "MFAR·Van." and "MFAR·Caus." are linked to adaptive-tanh implementations with K=1K=1, making their correspondence to the multi-frequency sinusoidal module in Eq. (2) unclear. The supplied code, data, and metadata also contain inconsistencies concerning network widths and planar-case identifiers.

The reporting conventions require reconciliation as well. For example, the bundled per-seed best errors for fixed TT at NC=64,R=32N_C=64,R=32 have mean 0.0509 and median 0.0277, whereas the corresponding Plane 1 entry in Table 2 is 0.028 and the caption states an average over three seeds. It remains unclear whether the table uses different runs or a different aggregation convention. These discrepancies make it difficult to identify the exact configurations and reporting procedures underlying the comparisons. The authors should clarify the relationship between the figure package and the main-table results, including the model variants, benchmark settings, seed aggregation, and checkpoint-selection conventions.

Questions for authors
  1. Could the authors clarify how the MFAR activation relates to the adaptive activation family in Jagtap et al., "Deep Kronecker neural networks," and identify the main additional contribution of MFAR-SVS? Which theoretical results or controlled comparisons most clearly demonstrate the added value of combining this activation with structured variable separation?

  2. How is the deterministic infinite-width claim in Step 3 of Appendix A.6 justified for the frequency scales and branch weights shared across hidden units in Eq. (31)? For fixed KK, the corresponding gradients are W−1/2W^{-1/2}-scaled sums, whose outer products need not concentrate to deterministic expectations solely because the width increases. Could the authors justify the stated limit under the given parameterization, or clarify how this claim should be revised?

  3. How should the network-output NTK spectral results be interpreted in relation to optimization of the PDE/initial/boundary residual objective in Eq. (13)? In the fixed/adaptive comparisons in Figure 5, is the fixed-frequency kernel constructed by freezing the frequency block at the same parameter state as the adaptive model, or are the kernels evaluated on separately trained models? Please distinguish the same-parameter spectral conclusion of Proposition 4 from claims about residual-loss convergence along training trajectories.

  4. Could the authors clarify how the nondecomposed adaptive variants in Figure 3 relate to the proposed MFAR architecture? The bundled plotting and model code maps the MFAR-Van. and MFAR-Caus. panels to adaptive-tanh implementations with K=1K=1, whereas Eq. (2) defines MFAR using sinusoidal branches with learnable frequency scales and branch weights. Are these implementations specific to Figure 3, or do they also underlie the corresponding results in the main tables? Please also reconcile the seed-aggregation and checkpoint conventions between Figure 3 and Table 2, specifying the seed sets, whether errors are means or medians across seeds, and whether they are final-checkpoint errors or minimum reference errors over evaluated checkpoints.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wSeq — Updated Flow Results

Thank you for the detailed review.

1. DKN and novelty (CW1, Q1). DKN's MFAR uses sinusoidal bases, αm=cm/K\alpha_m=c_m/K, and layer-wise sharing; we will cite it. Our contributions are grouped low-rank PINNs, PDE-constrained tangent-feature analysis, and controlled evaluation.

The activation is ϕ(z)=13∑m=13cmsin⁡(nmz)\phi(z)=\frac13\sum_{m=1}^3c_m\sin(n_mz), with scales initialized at n=(1/3,2/3,1)n=(1/3,2/3,1) and c=(1,1,1)c=(1,1,1) and optimized with network weights β\beta. Spectral parameters are shared within layers but independent across layers/subnetworks. The table uses the resulting Flow configuration.

2. PSD ordering (CW2). For the implemented weighted residual vector ρ=W1/2r\rho=W^{1/2}r, at identical numerical parameters, points, weights, and Jacobian scaling, define Hβ=∂βρH_\beta=\partial_\beta\rho, Hη=∂ηρH_\eta=\partial_\eta\rho. Then

Klearn=HβHβT+HηHηT,Kfrozen=HβHβT.K_{\rm learn}=H_\beta H_\beta^T+H_\eta H_\eta^T,\qquad K_{\rm frozen}=H_\beta H_\beta^T.

For any unit vector vv, the Rayleigh quotient increases by ∥HηTv∥22≥0\|H_\eta^Tv\|_2^2\geq0, implying λmin⁡(Klearn)≥λmin⁡(Kfrozen)\lambda_{\min}(K_{\rm learn})\geq\lambda_{\min}(K_{\rm frozen}), including zero eigenvalues and equality cases. We will remove unconditional conditioning/smallest-positive-eigenvalue claims; trajectory/frequency-learning gains do not follow.

3. Actual PINN kernel (CW2, Q3). For ri=Fi[uθ](zi)−yir_i=\mathcal F_i[u_\theta](z_i)-y_i, Jr=∂θrJ_r=\partial_\theta r, fixed points/weights, and known PDE coefficients,

∂pri=DFi[uθ](∂puθ)(zi),Kρ=W1/2JrJrTW1/2.\partial_pr_i=D\mathcal F_i[u_\theta](\partial_pu_\theta)(z_i),\qquad K_\rho=W^{1/2}J_rJ_r^TW^{1/2}.

The kernel includes initial/boundary and cross-block terms. For L=∥ρ∥2/2L=\|\rho\|^2/2, Euclidean gradient flow yields ρ˙=−Kρρ\dot\rho=-K_\rho\rho, with an additional same-state loss descent of −∥HηTρ∥2-\|H_\eta^T\rho\|^2. These results apply to a common fixed batch, without implying Adam, resampling, or solution-error guarantees.

The PDE tangent features for perturbation gg are: transport gt+b⋅∇gg_t+b\cdot\nabla g; Helmholtz Δg+κ2g\Delta g+\kappa^2g; Poisson Δ5g\Delta_5g; Klein–Gordon gtt−Δg+2uθgg_{tt}-\Delta g+2u_\theta g; and Burgers gt+(uθ⋅∇)g+(g⋅∇)uθ−νΔg\mathbf g_t+(\mathbf u_\theta\cdot\nabla)\mathbf g+(\mathbf g\cdot\nabla)\mathbf u_\theta-\nu\Delta\mathbf g.

For u=R−1/2∑rUrVru=R^{-1/2}\sum_rU_rV_r, ∂ηUu=R−1/2∑rVr∂ηUUr\partial_{\eta^U}u=R^{-1/2}\sum_rV_r\partial_{\eta^U}U_r. Shared-parameter sums and PDE couplings jointly determine the residual-kernel structure beyond the generic PSD identity.

4. Infinite width (CW2, Q2). The finite-width results do not require concentration of shared parameters. We will scope the analysis to these results and remove unsupported infinite-width/infinite-rank claims.

5. Figure 5 (Q3). Kernels use identical adaptive checkpoints and parameter values, with/without activation-parameter Jacobian coordinates. We will clarify the checkpoint, coordinate mask, points, scaling, kernel definition, and model version so that the same-state comparison is explicit.

6. Figure 3/reporting (CW3, Q4). The questioned panels used tanh-K1, not the stated sinusoidal model. We will correct this label; any replacement panel must use a verified run of the stated model. Fixed-TT's mean best error is 0.050908 (median 0.027716); 0.028 is the median. We will distinguish aggregation, best/final errors, and model versions, and check settings against code and logs. No replacement result is claimed here; Flow results cannot replace the planar figure.

Flow best relative L2L^2 errors. Bold: lowest mean per Eq/NC across ranks. Low-rank: adaptive sinusoidal activations and network weights. NC128 non-decomposition baselines unavailable.

ModelNCREq1Eq2
Vanilla64—9.1e-2±7.8e-31.1e-1±1.9e-2
Causal64—8.7e-2±7.8e-39.8e-2±5.1e-3
Causal Attn.64—1.0e-1±9.1e-31.3e-1±2.2e-2
MFF64—2.9e-2±1.4e-27.1e-2±1.2e-2
IFeF64—5.5e-2±9.1e-37.6e-2±1.1e-2
NeuSA64—4.58e-2±3.33e-36.60e-2±7.37e-3
CP64324.5e-2±2.9e-27.5e-2±3.7e-2
TT64325.1e-2±1.9e-27.5e-2±3.4e-2
Tucker64325.8e-2±2.8e-27.6e-2±3.2e-2
SVS64321.6e-2±1.4e-23.7e-2±1.7e-2
CP64644.4e-2±2.8e-27.5e-2±3.8e-2
TT64645.0e-2±2.0e-26.6e-2±3.4e-2
Tucker64645.6e-2±2.4e-26.9e-2±3.3e-2
SVS64641.9e-2±1.4e-24.6e-2±1.7e-2
CP128322.5e-3±2.0e-48.4e-3±7.5e-4
TT128322.5e-3±3.1e-47.5e-3±7.5e-4
Tucker128322.5e-3±4.5e-47.5e-3±1.1e-3
SVS128322.0e-3±2.4e-45.7e-3±2.6e-4
CP128642.2e-3±3.1e-47.3e-3±9.2e-4
TT128643.2e-3±6.3e-48.5e-3±2.2e-3
Tucker128642.8e-3±5.6e-49.4e-3±4.2e-3
SVS128642.1e-3±2.4e-45.9e-3±1.4e-4

All reported mean±sample SD values are arithmetic over seeds 445/987/123. Low-rank uses full-collocation Adam 1e-3, 150010 updates, with the best reference error recorded every 5k. Other baselines use mini-batch cosine training and select every 10k; budgets are 150k except MFF Eq2 (70k) and Causal Attn. (140k). Baselines select on 64364^3 and report 1283128^3 reference error; low-rank selects on 1283128^3.

Thank you for the detailed responses and clarifications. I appreciate that the authors have acknowledged several of my concerns, particularly regarding the infinite-width NTK analysis and the experimental implementation issues.

However, I maintain my original rating of 2 (Weak Reject) for the following reasons.

  1. Theoretical contribution: Removing the unsupported infinite-width and infinite-rank claims is appropriate, but the remaining PSD decomposition is a fairly general consequence of adding trainable parameter directions, rather than a result specific to adaptive frequency learning. The newly introduced PDE-specific tangent features do not yet establish a clear optimization advantage for the residual kernel.

  2. Methodological novelty: Although the relationship between MFAR and existing adaptive activation methods is now clarified, the novelty and practical advantages of SVS over existing separable and grouped factorization approaches, such as SPINN and TensoRF, remain insufficiently established.

  3. Experimental evidence: The authors acknowledge the incorrect activation implementation in Figure 3 and the mean/median reporting discrepancy, but verified replacement results for the affected planar experiments are still missing. The proposed additional ablations are also not yet supported by results.

Overall, I appreciate the authors' efforts, but the remaining theoretical, methodological, and experimental concerns prevent me from changing my initial assessment.

AuthorsOpenReview ↗

Response to Reviewers wSeq (Round 2)

We thank the reviewer for the follow-up. The revised manuscript directly addresses the three remaining concerns through new theoretical results, solution-structure analysis, and completed, verified experiments—not merely clarifications of earlier claims.

1. Theoretical contribution: beyond the general PSD identity (CW2, Q2–Q3).

Section 4 has been reorganized around a central question: how does activation learning change the optimization directions available for reducing PDE-constrained residuals?

We derive finite-width tangent features for grouped neural factors, accounting for shared-parameter interactions, and propagate them through PDE/IC/BC operators to obtain the weighted residual kernel. For residual ρ\rho, the additional instantaneous gradient-flow descent is ∥Hc⊤ρ∥2\|H_c^\top\rho\|^2, establishing a residual-dependent characterization beyond the general PSD increment (Sections 4.1–4.3, Eq. (10)).

More importantly, Proposition 4 (Section 4.4) establishes an exact finite-step physical-loss reduction. With fixed branch scales, linear PDE constraints, and other parameters held fixed, the residual is affine in a final-layer activation mixing block. A damped block update yields strictly positive loss reduction whenever its block gradient is nonzero.

At each of 24 saved Flow states, we updated the spatial and temporal blocks separately. All 48 updates reduced physical loss on the collocation points used for the block solve. The predicted residual matched direct reevaluation to within 3×10−133\times10^{-13} relative discrepancy (Section 5.4 and Appendix A.5, Table 7).

2. Methodological contribution: solution-dependent variable grouping (CW1, Q1).

While SPINN and TensoRF provide precedents, our contribution is to characterize when grouping is advantageous through the unfolding-rank structure of PDE solutions.

For transport, SVS jointly represents (x,y)(x,y) and separates time. At rank RR, optimal unrestricted grouped approximation has error et(R)e_t(R), whereas coordinate-wise CP has error at least max⁡{ex(R),ey(R),et(R)}\max\{e_x(R),e_y(R),e_t(R)\}. Thus, grouping avoids additional spatial unfolding-rank restrictions while retaining low-rank spatial–temporal coupling.

For example, Uijk=TiMjkU_{ijk}=T_iM_{jk} with rank⁡(M)=q\operatorname{rank}(M)=q requires one grouped term but qq CP terms. At relative error 10−310^{-3}, Flow requires grouped ranks 6/8 versus CP lower bounds 11/17, and Gaussian requires 7/8 versus 12/17 (Section 3.2, Table 1, and Appendix B.3).

The unfolding ranks motivate keeping the two spatial coordinates in one factor. Tables 2–5 show how this choice performs at the training ranks R=32R=32 and 6464. SVS achieves the lowest mean among four factorizations in 6/8 Flow, 5/8 Plane, and 8/8 Gaussian settings (Tables 2–4). Conversely, CP/TT performs better on the tested separable Helmholtz targets (Table 5), confirming the importance of solution structure.

Our contribution therefore connects PDE solution structure, representation constraints, and empirical performance to guide factorization design.

3. Experimental reliability: completed corrections and controlled evidence (CW3, Q4).

First, Table 3 reports the corrected Plane results. It covers 44 three-seed configurations (132 runs), with mean ± sample SD and common-grid checkpoint selection. Appendix B.8 documents the mean/median correction and withdrawal of the tanh-K1 panels, entries and timings. The supplementary code covers both Plane equations and all ten models, including the single-sine Vanilla and Causal baselines. It provides training and timing scripts for a common 64364^3 collocation grid, with checkpoint evaluation on a 1283128^3 reference grid.

Second, we completed the matched Flow ablation. We compared fixed and learned activations within CP, TT, Tucker, and SVS using the same initialization, collocation points, objective, optimizer, training budget, and checkpoint rule for each pair. At NC=128N_C=128, activation learning lowered the three-seed mean relative L2L^2 error in all 16 configurations in Table 6 (Section 5.3).

Third, the evaluation has been extended. Table 5 includes tenfold-frequency Helmholtz experiments, while Appendix B.10 (Table 16) adds nonlinear viscous Burgers equations, demonstrating performance across different PDE structures rather than only favorable transport cases.

In summary, the three concerns underlying the original rating are now addressed by substantive evidence: (i) an exact conditional PDE-residual loss-reduction result beyond the PSD identity, (ii) a rank-based explanation of SVS's advantages and limitations, and (iii) corrected Plane results and completed controlled ablations. We respectfully encourage the reviewer to reconsider the assessment in light of these new theoretical and experimental findings.

Reviewer_V5P2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MFAR-SVS, a physics-informed neural network (PINN) architecture that focus on the common failure mode of PINNs, the spectral bias and the coupling among variables representations when training on high-dimensional PDEs. It combines adaptive multi-frequency representations (MFAR) with structured variable separation (SVS) for spatiotemporal representation. In MFAR, each neural factor combines multiple sinusoidal branches and their frequencies and weights. Then the solution is represented through SVS, which groups strongly correlated spatial variables and separates them from the temporal variable, preserving unrestricted interactions within the spatial group while imposing low-rank coupling across groups. The authors further use neural tangent kernel (NTK) theory to illustrate the behavior of the network in the theoretical infinite-width and infinite-rank limits. The benchmark results show that the method has a considerable advantage over baselines.

Critical strengths
  1. The paper is well-written with clear contributions. Two components (MFAR and SVS) in the algorithm are presented clearly and are easy to follow.

  2. Extending based on SIREN, MFAR introduces multi-branch sinusoidal activations with learnable frequencies and weights inside each activation, which improves on SIREN's drawback on prior set frequencies.

  3. SVS provides a low-rank decoupling strategy by grouping coordinates, which preserves unrestricted intra-group interactions.

  4. The NTK analysis in this paper gives a deep understanding of the architecture and explains how the coordinate decomposition influences the kernel decomposition.

Critical weaknesses
  1. The abstract and introduction overemphasize the contribution of the NTK analysis. In Tancik et al. (2020), the NTK analysis of Fourier feature embeddings shows the Fourier embedding transforms the NTK of the neural network into a shift-invariant kernel whose eigenvalue decay can be controlled by the bandwidth σ\sigma. Therefore, the spectral bias can be alleviated by tuning σ\sigma. The paper does not discuss similar statements, and the claim of the closed-form NTK is too strong.

  2. The paper does not compare against recent SOTA methods or against methods for spectral bias. For example, many improvements over Causal PINN have since been proposed, such as [1]. MFAR method is based on SIREN and is contrasted with RFF-based methods, but no such baselines are included in the paper.

  3. The paper lacks detailed settings of each experiment or the configurations of the baselines, only some hyperparameters are given in Appendix B.1 Experimental Details. For the results, no standard deviations are reported. In addition, the training time of the proposed method is relatively long in Table 1 and 2. Moreover, the ablation in Table 7 does not show a substantial difference between fixed and learnable frequencies.

Questions for authors
  1. Could the authors explain why their NTK analysis demonstrates that the method mitigates spectral bias? Can the analysis have some similar derivation with paper for Tancik et al. (2020), talking about the eigenvalue decay of the network's NTK. If not, the claim in the abstract and introduction should not be that strong.

  2. Following Weakness 2, could the authors add comparisons with more recent SOTA methods, or with baselines specifically designed for spectral bias, such as the commonly used Fourier-based baselines, such as [2] and [3]?

  3. Could the authors provide more detailed experimental settings and report errors mean ±\pm std?

  4. The benchmarks include a high-dimensional problem (5D Poisson) and a coupled problem (Flow Mixing), but lack problems that exhibit high-frequency components. Could the authors add such experiments? A simple modification would be a high-frequency 3D Helmholtz problem obtained by multiplying a1a_1, a2a_2, and a3a_3 by 10.

[1] C. Zhao, X. Xie, and W. Chen. Causal Attention: Adaptive enforcement of causality in physics-informed neural networks. Journal of Computational Physics, 2026, 115071.

[2] S. Wang, H. Wang, and P. Perdikaris. On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021.

[3] Y. Wu, M. Aguiar, K. H. Johansson, and M. Barreau. Iterative training of physics-informed neural networks with Fourier-enhanced features. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer V5P2 — Updated Flow Results

Thanks for the suggestions on theory, comparisons, and reporting.

1. NTK and spectral bias (CW1, Q1). Let ρ=W1/2r\rho=W^{1/2}r collect weighted PDE and initial/boundary residuals, with ordinary parameters β\beta and adaptive activation parameters η\eta. Under identical parameters, collocation points, and loss weights, define Hβ=∂βρH_\beta=\partial_\beta\rho and Hη=∂ηρH_\eta=\partial_\eta\rho. Then

Klearn=Kfrozen+HηHηT,Kfrozen=HβHβT.K_{\rm learn}=K_{\rm frozen}+H_\eta H_\eta^T,\qquad K_{\rm frozen}=H_\beta H_\beta^T.

Consequently, Klearn⪰KfrozenK_{\rm learn}\succeq K_{\rm frozen} and λmin⁡(Klearn)≥λmin⁡(Kfrozen)\lambda_{\min}(K_{\rm learn})\geq\lambda_{\min}(K_{\rm frozen}). Beyond this identity, we derive PDE-specific constraint tangent features for transport, Helmholtz, Poisson, Klein–Gordon, and coupled Burgers. We will clarify the assumptions without claiming guaranteed spectral-bias mitigation or convergence acceleration.

2. Fourier/recent baselines (CW2, Q2). We compare complete methods rather than isolate activation effects. Flow best relative L2L^2 errors. Bold: lowest mean per Eq/NC across ranks. Low-rank: adaptive sinusoidal activations and network weights. NC128 non-decomposition baselines unavailable.

ModelNCREq1Eq2
Vanilla64—9.1e-2±7.8e-31.1e-1±1.9e-2
Causal64—8.7e-2±7.8e-39.8e-2±5.1e-3
Causal Attn.64—1.0e-1±9.1e-31.3e-1±2.2e-2
MFF64—2.9e-2±1.4e-27.1e-2±1.2e-2
IFeF64—5.5e-2±9.1e-37.6e-2±1.1e-2
NeuSA64—4.58e-2±3.33e-36.60e-2±7.37e-3
CP64324.5e-2±2.9e-27.5e-2±3.7e-2
TT64325.1e-2±1.9e-27.5e-2±3.4e-2
Tucker64325.8e-2±2.8e-27.6e-2±3.2e-2
SVS64321.6e-2±1.4e-23.7e-2±1.7e-2
CP64644.4e-2±2.8e-27.5e-2±3.8e-2
TT64645.0e-2±2.0e-26.6e-2±3.4e-2
Tucker64645.6e-2±2.4e-26.9e-2±3.3e-2
SVS64641.9e-2±1.4e-24.6e-2±1.7e-2
CP128322.5e-3±2.0e-48.4e-3±7.5e-4
TT128322.5e-3±3.1e-47.5e-3±7.5e-4
Tucker128322.5e-3±4.5e-47.5e-3±1.1e-3
SVS128322.0e-3±2.4e-45.7e-3±2.6e-4
CP128642.2e-3±3.1e-47.3e-3±9.2e-4
TT128643.2e-3±6.3e-48.5e-3±2.2e-3
Tucker128642.8e-3±5.6e-49.4e-3±4.2e-3
SVS128642.1e-3±2.4e-45.9e-3±1.4e-4

All reported mean±sample SD values are arithmetic over seeds 445/987/123. Low-rank uses full-collocation Adam 1e-3, 150010 updates, with the best reference error recorded every 5k. Other baselines use mini-batch cosine training and select every 10k; budgets are 150k except MFF Eq2 (70k) and Causal Attn. (140k). Baselines select on 64364^3 and report 1283128^3 reference error; low-rank selects on 1283128^3.

Results. At NC64, SVS has the lowest tabulated mean errors on both equations; relative to the updated NeuSA means, the differences are 65.3% for Eq1 and 44.7% for Eq2. Protocols and seed variability differ, so this comparison is descriptive and does not establish statistical superiority. At NC128, SVS remains competitive and achieves the lowest error on Eq2. MFF comparisons cannot isolate activation effects.

3. Configuration/statistics (CW3, Q3). The model uses ϕ(z)=13∑m=13cmsin⁡(nmz)\phi(z)=\frac13\sum_{m=1}^3c_m\sin(n_mz) with adaptive activation parameters optimized with network weights. These parameters are shared within each hidden layer and learned independently across layers/factors. Each factor has width 128 and two hidden layers; SVS groups tt and (x,y)(x,y).

We will clarify parameter/feature counts, Fourier conventions, loss weights, sampling/refresh and selection protocols, report arithmetic mean±sample SD, distinguish best/final errors, and correct aggregation discrepancies.

4. Cost and Table 7 (CW3). Parameter/update counts alone do not establish efficiency; efficiency claims require matched runtime, memory, and time-to-error evidence. Table 7 does not isolate individual activation effects or establish consistent gains. We will add a matched four-arm ablation—freeze both n,cn,c, learn nn only, learn cc only, or learn both—using identical initialization, architecture, collocation points, optimizer, budget, and seeds, and report best/final errors with mean±sample SD.

5. Tenfold Helmholtz frequencies (Q4). A tenfold-frequency extension of Eq1=(2,2,1) would use (20,20,10). For Δu+κ2u=q\Delta u+\kappa^2u=q, an analytic setting is u∗=sin⁡(20πx1)sin⁡(20πx2)sin⁡(10πx3)u_*=\sin(20\pi x_1)\sin(20\pi x_2)\sin(10\pi x_3), with q=(κ2−900π2)u∗q=(\kappa^2-900\pi^2)u_* and homogeneous Dirichlet conditions. This specifies a possible stress test, not a completed numerical result or an extension of the present empirical claims.

Thank you for the response and the additional experimental results. The NTK analysis was presented as one of the main contributions of the paper, but after the rebuttal the theoretical claims are substantially weakened in mitigating spectral bias. The structure of the therotical part needs to be reconsidered. Therefore I will maintain my rating.

AuthorsOpenReview ↗

Response to Reviewers V5P2 (Round 2)

We thank the reviewer for highlighting the theoretical contribution as the remaining concern. We agree that the original NTK interpretation was overly broad. The revised theory is not simply a weakened spectral-bias claim; it addresses a more precise question: how do activation learning and structured factorization affect PDE-residual reduction? We have reorganized Section 4 accordingly.

1. Reframed theoretical contribution (latest concern, CW1/Q1).

Section 4 develops a finite-width, PDE-specific analysis:

  • Section 4.1: Derives the output tangent kernel (Eq. (4)), accounting for factor interactions and shared activation parameters.
  • Section 4.2: Applies PDE/IC/BC operators to obtain weighted residual kernels (Eqs. (5)–(6)). For transport, Eq. (7) explicitly incorporates temporal and spatial factor derivatives.
  • Section 4.3: Compares learned and frozen activations at identical parameter states (Proposition 3, Eq. (8)): Klearn=Kfrozen+HcHc⊤.K_{\mathrm{learn}}=K_{\mathrm{frozen}}+H_cH_c^\top. Beyond this general PSD identity, the additional instantaneous gradient-flow descent is ∥Hc⊤ρ∥2\|H_c^\top\rho\|^2 (Eq. (10)), connecting activation tangent features to the physical residual.
  • Section 4.4: Establishes a conditional finite-step residual-loss reduction.

Unlike Tancik et al.'s stationary Fourier-feature kernel analysis, our theory characterizes finite-width, state-dependent PDE-residual optimization with structured factors and shared activation parameters. We no longer claim universal spectral-bias mitigation or accelerated convergence, and have revised the abstract and introduction accordingly.

2. Exact finite-step result and verification.

Proposition 4 (Section 4.4, Eq. (11), Appendix A.5) establishes an exact conditional physical-loss reduction when optimizing the final hidden-layer activation mixing coefficients of one factor, with fixed frequencies, linear constraints, and all other parameters held fixed. The reduction is strictly positive when the block gradient is nonzero.

At 24 saved Flow states, separate spatial and temporal block updates reduced physical loss in all 48 cases (Section 5.4 and Appendix A.5, Table 7). Direct reevaluation matched the affine prediction within 3×10−133\times10^{-13} relative discrepancy. This validates the block-level result, without implying convergence guarantees for full Adam training.

3. Controlled activation-learning evidence (CW2–3).

The original model allowed the branch scales nkn_k to vary. In a single-seed pilot starting at n=(1,2,3)n=(1,2,3), we trained either ckc_k or nkn_k within the activation. For TT on both Flow equations, training ckc_k gave a smaller standard deviation of relative error over the last six checkpoints. This pattern varied by factorization. This motivated our revised model, which fixes nkn_k and learns each frequency branch's contribution through its mixing coefficient ckc_k.

Both arms of the matched Flow ablation at NC=128N_C=128 used the same scales (1/3,2/3,1)(1/3,2/3,1) (Section 5.3, Table 6). Learning the activation yielded a lower relative L2L^2 error than keeping it fixed in all 48 same-seed comparisons across CP, TT, Tucker, and SVS (16 configurations, three seeds each).

4. Stronger baselines and reproducibility (CW2–3).

Tables 2–4 include Vanilla, Causal, Causal Attention, MFF, IFeF, and NeuSA. SVS achieves the lowest mean among four factorizations in 6/8 Flow, 5/8 Plane, and 8/8 Gaussian settings. Appendix B.10 (Table 16) adds nonlinear viscous Burgers, where the tested SVS configuration is not the strongest.

Section 5.1 and Appendix B.1 clarify architecture, sampling, schedules, seeds, evaluation, and checkpoint selection. Tables 2–5 report mean ± sample SD over three seeds. Timing estimates (Appendix B.1) and common-grid update comparisons (Section 5.5 and Appendix B.3, Table 12) are distinguished to avoid unsupported efficiency claims.

5. Completed tenfold-frequency Helmholtz experiment (Q4).

Table 5 (Section 5.2) compares the original (2,2,1)(2,2,1) target with the requested (20,20,10)(20,20,10) target. All 114 registered runs completed 150,000 Adam updates, and best checkpoints were reevaluated on a common 1283128^3 grid.

On the high-frequency target, TT (rank 32) achieves the lowest mean at NC=64N_C=64 (0.18±0.0610.18\pm0.061), while CP (rank 64) leads at NC=128N_C=128 (0.0024±0.000280.0024\pm0.00028). SVS performs less favorably on this separable target, illustrating the dependence of factorization performance on solution structure.

In summary, Section 4 has been fundamentally reorganized around three contributions: (i) finite-width PDE-residual kernels for structured factors, (ii) residual-dependent activation-learning directions, and (iii) an exact conditional finite-step loss-reduction result. Together with the new experiments and controlled ablations, these revisions provide a more rigorous and verifiable theoretical foundation that directly addresses the reviewer's concern.

Reviewer_hUH9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of approximating PDE solutions using PINNs. It replaces SIREN's sinusoidal activation with a sum of sinusoids with learnable frequencies. A similar activation was proposed in STAF [1] for INRs.

Critical strengths
  • Studying sinusoidal networks for PINNs is an important research direction.

  • Fixed/adaptive frequency comparisons across factorizations help assess the contributions of MFAR and SVS (Tables 1–4), complemented by an ablation on the number of frequencies (Table 7).

Critical weaknesses
  • STAF [1] is not cited, and the differences from it need clarification.

  • SVS closely follows SPINN's separable formulation, replacing individual coordinates with variable groups. The novelty of this extension needs clarification.

  • Most benchmark PDEs are linear; more demanding nonlinear benchmarks would strengthen the evaluation. For examples, see NeuSA [2], a recent spectral PINN paper that is neither cited nor compared against.

Questions for authors
  • Even with a frozen first layer, SIREN's hidden weights change harmonic amplitudes, as explained in TUNER [3]. Please clarify the claim of fixed frequency content.

  • The discussion of sinusoidal networks is not up to date, omitting STAF, TUNER, SASNet [4], and PDE applications such as NISE [5].

  • Clarify SVS's novelty relative to SPINN's coordinate-wise separation and the grouped factorization in TensoRF [6].

  • Figure 1 is dense and difficult to follow. The third column uses notation that has not yet been introduced.

[1] Morsali et al. "A unified theory of sinusoidal activation families for implicit neural representations." TMLR. 2026.

[2] Bizzi et al. "Neuro-spectral architectures for causal physics-informed networks." NeurIPS. 2025.

[3] Novello et al. "Tuning the frequencies: Robust training for sinusoidal neural networks." CVPR. 2025.

[4] Feng et al. "SASNet: Spatially-adaptive sinusoidal networks for INRs." CVPR. 2026.

[5] Novello et al. "Neural implicit surface evolution." ICCV. 2023.

[6] Chen et al. "TensoRF: Tensorial radiance fields." ECCV. 2022.

1 comment
AuthorsOpenReview ↗

Response to Reviewer hUH9 — Final version

Clarifying Methodological Contributions, Theoretical Soundness, and Experimental Evidence

We thank the reviewer for the constructive comments. We have revised the manuscript to clarify our contributions, strengthen the PDE-specific analysis, and expand the experiments. Our key advance is not the combination itself, but the theoretical and empirical characterization of when and why these components benefit PDE-constrained learning.

1. MFAR versus STAF/DKN (CW1).

The activation in Eq. (1) is a zero-phase special case of STAF and also falls within the DKN family (Sections 2.1 and 3.1). The three sinusoidal branches share an affine input and have prescribed scales (1/3,2/3,1)(1/3,2/3,1). Each hidden layer learns one vector of mixing coefficients shared across its units (Appendix B.1).

Our contribution lies in characterizing activation learning under PDE constraints, rather than proposing a new activation formula. Sections 4.1–4.2 derive finite-width output and weighted PDE/IC/BC residual kernels, accounting for shared activation parameters and factor interactions. Beyond the general PSD kernel increment, Proposition 4 (Section 4.4, Eq. (11)) establishes an exact conditional residual-loss reduction for a final-layer activation mixing block under fixed linear PDE constraints, identifying when activation learning provides an effective descent direction. Section 5.3 reports the matched Flow ablation.

2. SVS versus SPINN/TensoRF (CW2, Q3).

We acknowledge SPINN and TensoRF as precedents (Sections 2.2 and 3.2). Our contribution is a PDE-solution-driven analysis identifying when grouping removes restrictive rank constraints and explaining its advantages and limitations.

For transport, SVS jointly models (x,y)(x,y) while separating time. At rank RR, unrestricted grouped approximation achieves optimal error et(R)e_t(R), whereas coordinate-wise CP has error at least max⁡{ex(R),ey(R),et(R)}\max\{e_x(R),e_y(R),e_t(R)\}. Thus, grouping avoids additional spatial unfolding-rank restrictions. For example, Uijk=TiMjkU_{ijk}=T_iM_{jk} with rank⁡(M)=q\operatorname{rank}(M)=q requires one grouped term but qq CP terms.

At relative error 10−310^{-3}, Flow requires grouped ranks 6/8 versus CP lower bounds 11/17; Gaussian requires 7/8 versus 12/17 (Section 3.2, Table 1, and Appendix B.3). At matched collocation density and rank, SVS achieves the lowest mean relative error among the four factorizations in 6/8 Flow, 5/8 Plane, and 8/8 Gaussian settings (Tables 2–4), whereas CP/TT performs better on separable Helmholtz problems (Table 5).

These results establish a solution-dependent rationale for grouping, rather than claiming universal superiority. The rank analysis concerns unrestricted approximation; neural performance also depends on capacity and optimization.

3. Nonlinear PDEs and stronger baselines (CW3).

Appendix B.10 (Table 16) adds coupled 2D viscous Burgers equations (ν=0.01\nu=0.01) with periodic boundaries, comparing four factorizations against Vanilla, MFF, IFeF, and NeuSA. At NC=128N_C=128, Tucker achieves 6.4×10−46.4\times10^{-4} versus 1.1×10−21.1\times10^{-2} for SVS, showing that nonlinear coupling does not necessarily favor grouping. We identify this as a single-seed supplementary experiment with a numerical reference and avoid broad generalizations.

NeuSA is also included in the main transport comparisons (Tables 2–4), while Table 5 evaluates tenfold-wavenumber Helmholtz equations.

4. SIREN spectrum and related literature (Q1–Q2).

We corrected the fixed-frequency characterization: prescribed SIREN activation scales do not imply a fixed network spectrum, since trainable affine maps and layer composition alter harmonic content, as explained by TUNER. Section 2.1 now discusses STAF, TUNER, SASNet, and NISE and clarifies their relationships to our work.

5. Figure 1 and controlled validation (Q4, CW1–3).

We expanded the caption of Figure 1 (Section 1) to define the factorization, activation parameters, and optimization notation.

In the matched Flow ablation at NC=128N_C=128 (Section 5.3, Table 6), learning the activation yielded a lower relative L2L^2 error than keeping it fixed in all 48 same-seed comparisons across CP, TT, Tucker, and SVS (16 configurations, three seeds each).

In summary, our contributions are (i) a PDE-solution-dependent analysis of grouping and rank restrictions, (ii) finite-width PDE-residual analysis and conditional loss reduction for activation learning, and (iii) controlled evidence of their complementary, task-dependent benefits. We have also clarified the theoretical assumptions and experimental limitations to address the soundness concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.