- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MFAR-SVS, a representation framework for physics-informed neural networks that combines adaptive multi-frequency sinusoidal activations with grouped low-rank variable separation. MFAR uses a weighted sum of sinusoidal branches with trainable frequency scales and mixing coefficients. SVS partitions the input variables into two predefined groups and combines the outputs of two subnetworks through a low-rank sum of products, allowing variables within each group, such as spatial coordinates, to be modeled jointly.
The paper presents a universal approximation result and an NTK decomposition for the grouped representation. It also analyzes the positive-semidefinite contribution of additional trainable parameters to the NTK under a comparison at the same parameter values. Experiments compare activation choices and factorization architectures on transport and Helmholtz problems, with additional Klein-Gordon and five-dimensional Poisson results in the appendix. The reported results suggest benefits from adaptive frequencies and spatial grouping on several transport problems, while some dimension-wise factorizations achieve lower errors on the separable Helmholtz benchmark.
Critical strengths
1. Clear and relevant motivation.
The paper addresses frequency representation and cross-variable coupling within a single framework. The two components, adaptive sinusoidal activations and grouped variable separation, correspond naturally to these challenges, making the motivation and overall architectural design easy to follow.
2. Useful empirical comparisons across architectures and problem settings.
The experiments consider several PDE families and compare fixed and adaptive frequency variants across multiple factorization architectures, ranks, and collocation densities. The reported improvements across different decomposition families suggest that adaptive multi-frequency representations can be useful across different factorization architectures, with particularly promising results on the transport problems. The paper also reports cases in which dimension-wise factorizations outperform SVS, particularly on the separable Helmholtz benchmark, helping clarify the scope and limitations of structured variable separation.
3. Supplementary code and data facilitate inspection of Figure 3.
The supplement provides plotting and training/model code, saved error fields, and per-seed metrics for the decomposed models in Figure 3. These materials provide a useful basis for inspecting the supplied implementations, checking how Figure 3 was generated, and examining the associated experimental settings.
Critical weaknesses
1. The methodological novelty is insufficiently established.
Jagtap et al., "Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions" (arXiv:2105.09513), already consider activations of the form with trainable coefficients and scales. Choosing sinusoidal basis functions, , and recovers the MFAR activation form, including the layer-wise parameter sharing used in the supplementary implementation. This limits the novelty of the activation itself. Combining this activation family with grouped low-rank representations is a reasonable design, and the component comparisons provide evidence of its utility. However, the paper should more clearly distinguish this architectural combination from a new activation method and identify which findings constitute its main additional methodological contribution.
2. The optimization interpretation is not fully justified, and the infinite-width argument contains a gap.
The finite-width NTK decomposition and the eigenvalue monotonicity in Proposition 4 are valid under their stated assumptions. However, the PSD increment is a general consequence of adding trainable parameter directions at a common parameter state. It does not, by itself, provide an explanation specific to frequency adaptation or establish an ordering between kernels along independently optimized training trajectories.
The connection to PINN optimization also requires more careful qualification. Even under gradient flow for the squared PDE/initial/boundary residual objective, the relevant kernel is formed from the Jacobian of the stacked, appropriately weighted residuals. The reported output-NTK spectra alone therefore do not establish improved convergence for this residual objective. The authors should clarify the scope of the optimization interpretation and distinguish the kernel comparison at a common parameter state from the training behavior it is intended to explain.
There is also a specific issue in Appendix A.6, Step 3, lines 931-934. Under Eq. (31), the frequency scales and branch weights are shared across hidden units. For fixed , their gradients are -scaled sums whose outer products can retain non-vanishing random fluctuations as width increases. The asserted convergence to deterministic expectations through a law-of-large-numbers argument is therefore not justified as written. This step requires correction, although it does not invalidate the finite-width PSD identity.
3. Several experimental specifications and reported quantities require reconciliation.
In the Figure 3 supplementary package, the nondecomposed adaptive panels labeled "MFAR·Van." and "MFAR·Caus." are linked to adaptive-tanh implementations with , making their correspondence to the multi-frequency sinusoidal module in Eq. (2) unclear. The supplied code, data, and metadata also contain inconsistencies concerning network widths and planar-case identifiers.
The reporting conventions require reconciliation as well. For example, the bundled per-seed best errors for fixed TT at have mean 0.0509 and median 0.0277, whereas the corresponding Plane 1 entry in Table 2 is 0.028 and the caption states an average over three seeds. It remains unclear whether the table uses different runs or a different aggregation convention. These discrepancies make it difficult to identify the exact configurations and reporting procedures underlying the comparisons. The authors should clarify the relationship between the figure package and the main-table results, including the model variants, benchmark settings, seed aggregation, and checkpoint-selection conventions.
Questions for authors
-
Could the authors clarify how the MFAR activation relates to the adaptive activation family in Jagtap et al., "Deep Kronecker neural networks," and identify the main additional contribution of MFAR-SVS? Which theoretical results or controlled comparisons most clearly demonstrate the added value of combining this activation with structured variable separation?
-
How is the deterministic infinite-width claim in Step 3 of Appendix A.6 justified for the frequency scales and branch weights shared across hidden units in Eq. (31)? For fixed , the corresponding gradients are -scaled sums, whose outer products need not concentrate to deterministic expectations solely because the width increases. Could the authors justify the stated limit under the given parameterization, or clarify how this claim should be revised?
-
How should the network-output NTK spectral results be interpreted in relation to optimization of the PDE/initial/boundary residual objective in Eq. (13)? In the fixed/adaptive comparisons in Figure 5, is the fixed-frequency kernel constructed by freezing the frequency block at the same parameter state as the adaptive model, or are the kernels evaluated on separately trained models? Please distinguish the same-parameter spectral conclusion of Proposition 4 from claims about residual-loss convergence along training trajectories.
-
Could the authors clarify how the nondecomposed adaptive variants in Figure 3 relate to the proposed MFAR architecture? The bundled plotting and model code maps the MFAR-Van. and MFAR-Caus. panels to adaptive-tanh implementations with , whereas Eq. (2) defines MFAR using sinusoidal branches with learnable frequency scales and branch weights. Are these implementations specific to Figure 3, or do they also underlie the corresponding results in the main tables? Please also reconcile the seed-aggregation and checkpoint conventions between Figure 3 and Table 2, specifying the seed sets, whether errors are means or medians across seeds, and whether they are final-checkpoint errors or minimum reference errors over evaluated checkpoints.
3 comments
Thank you for the detailed responses and clarifications. I appreciate that the authors have acknowledged several of my concerns, particularly regarding the infinite-width NTK analysis and the experimental implementation issues.
However, I maintain my original rating of 2 (Weak Reject) for the following reasons.
-
Theoretical contribution: Removing the unsupported infinite-width and infinite-rank claims is appropriate, but the remaining PSD decomposition is a fairly general consequence of adding trainable parameter directions, rather than a result specific to adaptive frequency learning. The newly introduced PDE-specific tangent features do not yet establish a clear optimization advantage for the residual kernel.
-
Methodological novelty: Although the relationship between MFAR and existing adaptive activation methods is now clarified, the novelty and practical advantages of SVS over existing separable and grouped factorization approaches, such as SPINN and TensoRF, remain insufficiently established.
-
Experimental evidence: The authors acknowledge the incorrect activation implementation in Figure 3 and the mean/median reporting discrepancy, but verified replacement results for the affected planar experiments are still missing. The proposed additional ablations are also not yet supported by results.
Overall, I appreciate the authors' efforts, but the remaining theoretical, methodological, and experimental concerns prevent me from changing my initial assessment.
Response to Reviewers wSeq (Round 2)
We thank the reviewer for the follow-up. The revised manuscript directly addresses the three remaining concerns through new theoretical results, solution-structure analysis, and completed, verified experiments—not merely clarifications of earlier claims.
1. Theoretical contribution: beyond the general PSD identity (CW2, Q2–Q3).
Section 4 has been reorganized around a central question: how does activation learning change the optimization directions available for reducing PDE-constrained residuals?
We derive finite-width tangent features for grouped neural factors, accounting for shared-parameter interactions, and propagate them through PDE/IC/BC operators to obtain the weighted residual kernel. For residual , the additional instantaneous gradient-flow descent is , establishing a residual-dependent characterization beyond the general PSD increment (Sections 4.1–4.3, Eq. (10)).
More importantly, Proposition 4 (Section 4.4) establishes an exact finite-step physical-loss reduction. With fixed branch scales, linear PDE constraints, and other parameters held fixed, the residual is affine in a final-layer activation mixing block. A damped block update yields strictly positive loss reduction whenever its block gradient is nonzero.
At each of 24 saved Flow states, we updated the spatial and temporal blocks separately. All 48 updates reduced physical loss on the collocation points used for the block solve. The predicted residual matched direct reevaluation to within relative discrepancy (Section 5.4 and Appendix A.5, Table 7).
2. Methodological contribution: solution-dependent variable grouping (CW1, Q1).
While SPINN and TensoRF provide precedents, our contribution is to characterize when grouping is advantageous through the unfolding-rank structure of PDE solutions.
For transport, SVS jointly represents and separates time. At rank , optimal unrestricted grouped approximation has error , whereas coordinate-wise CP has error at least . Thus, grouping avoids additional spatial unfolding-rank restrictions while retaining low-rank spatial–temporal coupling.
For example, with requires one grouped term but CP terms. At relative error , Flow requires grouped ranks 6/8 versus CP lower bounds 11/17, and Gaussian requires 7/8 versus 12/17 (Section 3.2, Table 1, and Appendix B.3).
The unfolding ranks motivate keeping the two spatial coordinates in one factor. Tables 2–5 show how this choice performs at the training ranks and . SVS achieves the lowest mean among four factorizations in 6/8 Flow, 5/8 Plane, and 8/8 Gaussian settings (Tables 2–4). Conversely, CP/TT performs better on the tested separable Helmholtz targets (Table 5), confirming the importance of solution structure.
Our contribution therefore connects PDE solution structure, representation constraints, and empirical performance to guide factorization design.
3. Experimental reliability: completed corrections and controlled evidence (CW3, Q4).
First, Table 3 reports the corrected Plane results. It covers 44 three-seed configurations (132 runs), with mean ± sample SD and common-grid checkpoint selection. Appendix B.8 documents the mean/median correction and withdrawal of the tanh-K1 panels, entries and timings. The supplementary code covers both Plane equations and all ten models, including the single-sine Vanilla and Causal baselines. It provides training and timing scripts for a common collocation grid, with checkpoint evaluation on a reference grid.
Second, we completed the matched Flow ablation. We compared fixed and learned activations within CP, TT, Tucker, and SVS using the same initialization, collocation points, objective, optimizer, training budget, and checkpoint rule for each pair. At , activation learning lowered the three-seed mean relative error in all 16 configurations in Table 6 (Section 5.3).
Third, the evaluation has been extended. Table 5 includes tenfold-frequency Helmholtz experiments, while Appendix B.10 (Table 16) adds nonlinear viscous Burgers equations, demonstrating performance across different PDE structures rather than only favorable transport cases.
In summary, the three concerns underlying the original rating are now addressed by substantive evidence: (i) an exact conditional PDE-residual loss-reduction result beyond the PSD identity, (ii) a rank-based explanation of SVS's advantages and limitations, and (iii) corrected Plane results and completed controlled ablations. We respectfully encourage the reviewer to reconsider the assessment in light of these new theoretical and experimental findings.
Response to Reviewer wSeq — Updated Flow Results
Thank you for the detailed review.
1. DKN and novelty (CW1, Q1). DKN's MFAR uses sinusoidal bases, αm=cm/K, and layer-wise sharing; we will cite it. Our contributions are grouped low-rank PINNs, PDE-constrained tangent-feature analysis, and controlled evaluation.
The activation is ϕ(z)=31∑m=13cmsin(nmz), with scales initialized at n=(1/3,2/3,1) and c=(1,1,1) and optimized with network weights β. Spectral parameters are shared within layers but independent across layers/subnetworks. The table uses the resulting Flow configuration.
2. PSD ordering (CW2). For the implemented weighted residual vector ρ=W1/2r, at identical numerical parameters, points, weights, and Jacobian scaling, define Hβ=∂βρ, Hη=∂ηρ. Then
Klearn=HβHβT+HηHηT,Kfrozen=HβHβT.
For any unit vector v, the Rayleigh quotient increases by ∥HηTv∥22≥0, implying λmin(Klearn)≥λmin(Kfrozen), including zero eigenvalues and equality cases. We will remove unconditional conditioning/smallest-positive-eigenvalue claims; trajectory/frequency-learning gains do not follow.
3. Actual PINN kernel (CW2, Q3). For ri=Fi[uθ](zi)−yi, Jr=∂θr, fixed points/weights, and known PDE coefficients,
∂pri=DFi[uθ](∂puθ)(zi),Kρ=W1/2JrJrTW1/2.
The kernel includes initial/boundary and cross-block terms. For L=∥ρ∥2/2, Euclidean gradient flow yields ρ˙=−Kρρ, with an additional same-state loss descent of −∥HηTρ∥2. These results apply to a common fixed batch, without implying Adam, resampling, or solution-error guarantees.
The PDE tangent features for perturbation g are: transport gt+b⋅∇g; Helmholtz Δg+κ2g; Poisson Δ5g; Klein–Gordon gtt−Δg+2uθg; and Burgers gt+(uθ⋅∇)g+(g⋅∇)uθ−νΔg.
For u=R−1/2∑rUrVr, ∂ηUu=R−1/2∑rVr∂ηUUr. Shared-parameter sums and PDE couplings jointly determine the residual-kernel structure beyond the generic PSD identity.
4. Infinite width (CW2, Q2). The finite-width results do not require concentration of shared parameters. We will scope the analysis to these results and remove unsupported infinite-width/infinite-rank claims.
5. Figure 5 (Q3). Kernels use identical adaptive checkpoints and parameter values, with/without activation-parameter Jacobian coordinates. We will clarify the checkpoint, coordinate mask, points, scaling, kernel definition, and model version so that the same-state comparison is explicit.
6. Figure 3/reporting (CW3, Q4). The questioned panels used tanh-K1, not the stated sinusoidal model. We will correct this label; any replacement panel must use a verified run of the stated model. Fixed-TT's mean best error is 0.050908 (median 0.027716); 0.028 is the median. We will distinguish aggregation, best/final errors, and model versions, and check settings against code and logs. No replacement result is claimed here; Flow results cannot replace the planar figure.
Flow best relative L2 errors. Bold: lowest mean per Eq/NC across ranks. Low-rank: adaptive sinusoidal activations and network weights. NC128 non-decomposition baselines unavailable.
All reported mean±sample SD values are arithmetic over seeds 445/987/123. Low-rank uses full-collocation Adam 1e-3, 150010 updates, with the best reference error recorded every 5k. Other baselines use mini-batch cosine training and select every 10k; budgets are 150k except MFF Eq2 (70k) and Causal Attn. (140k). Baselines select on 643 and report 1283 reference error; low-rank selects on 1283.