acceptodds
Under review as a conference paper at ICLR 2027

Checkable NTK Positivity and Finite-Width Gradient Descent for Scalar- and Vector-Valued PINNs with Strong-Form, Weak-Form, and Nonlocal Linear Constraints

Abstract

We give checkable positive-definiteness criteria for the limiting neural tangent kernel (NTK) and high-probability finite-width gradient-descent guarantees for scalar- or vector-valued physics-informed neural networks (PINNs) with linear constraints. The constraints may be strong-form differential rows of any fixed finite order, including coupled systems, with any linear initial or boundary conditions; weak-form residual and boundary functionals; or finite-measure nonlocal observations such as integral, nonlocal-diffusion, and Dirac-data rows, all in any dimension. For each class, positive definiteness of the limiting NTK is equivalent to a rank condition on a finite coefficient, functional, or moment matrix of the fixed design: a certificate computed before training that detects structural zero modes. The model hypotheses are those of an ordinary two-layer PINN, a smooth nonpolynomial activation with bounded symmetric initialization, and are met by standard choices such as tanh with uniform initialization. Given a certificate, explicit width and step-size conditions ensure, with high probability, that the empirical NTK retains at least half the limiting gap and that full-batch gradient descent decreases the training loss geometrically at every iteration. Controlled experiments check the certificates and the finite-width mechanism.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
physics-informed neural networks, neural tangent kernel, convergence analysis, overparameterization, gradient descent, spectral gap, partial differential equations, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_CWLsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when Physics-Informed Neural Networks (PINNs) can be guaranteed to train successfully using gradient descent. The main idea is to connect NTK positivity to the independence of the physical constraints.

The paper provides a way to check whether PINN constraints produce a positive NTK and then uses that condition to prove convergence of gradient descent for sufficiently wide two-layer PINNs with linear constraints.

Critical strengths

The core theoretical logic appears sound. The authors do not simply assume a positive NTK gap. They characterize the nullspace using the constraint structure and then transfer that positivity to finite width. That is the main theoretical contribution.

I also do not see an obvious hole in the weak-form nullspace argument.

Critical weaknesses

The paper has Proposition C.9 to bridge this with a finite-witness lower bound, so this isn't a logical hole. But the practical strength of the finite-width theorem depends on obtaining a meaningful λ0\lambda_0, not merely establishing rank.

The main theoretical concern is the gap between existential positivity characterization and operational checkability. For strong-form constraints the certificate is a direct finite rank test. For general weak/nonlocal constraints, however, positivity may be witnessed by a sufficiently large finite Taylor/moment truncation, while the paper explicitly notes that no a priori truncation degree is available. Thus it is unclear whether the advertised “checkable necessary-and-sufficient certificate” constitutes a terminating decision procedure, especially for singular designs. Moreover, the finite-width GD theorem requires a quantitative lower bound λ0\lambda_0, whereas structural full rank alone only establishes strict positivity. The authors should clarify exactly what can be computed in finite time and how a usable λ0\lambda_0 is obtained for the general classes.

Questions for authors

For the general weak-form and nonlocal settings, does the proposed certificate give a terminating decision procedure for both positive and singular designs? If not, the paper should distinguish a finite witness for positivity from a fully checkable necessary-and-sufficient certificate.

The main results rely on linear constraints, while many practical PINNs contain nonlinear PDE terms. Could the authors clarify which parts of the positivity/certificate framework could potentially extend to nonlinear constraints, and which parts fundamentally fail? The current discussion suggests that a deterministic limiting kernel may fail for genuinely nonlinear rows.

Theorem 4.2 provides explicit width conditions for the finite-width GD guarantee. Could the authors clarify how large these theoretical width requirements are for the PDE problems used in the experiments, and how they compare with the widths actually tested? The experiments explicitly note that they do not verify the constants in Eq. (9).

1 comment

Thank you for the careful review.

W1. A usable λ0\lambda_0. Theorem 4.2 needs only that λmin⁡(K(0)∞)\lambda_{\min}(\mathbf K_{(0)}^\infty) is positive, which is what the positive-definiteness certificate proves. Corollary 4.3 takes λ0=λmin⁡(K(0)∞)\lambda_0=\lambda_{\min}(\mathbf K_{(0)}^\infty), so full rank gives convergence for every width above a threshold explicit in λ0\lambda_0. No lower bound on λmin⁡\lambda_{\min} can hold across designs, because it depends on the PDE, the collocation points, the activation. For a given design it is the smallest eigenvalue of an explicit N×NN\times N matrix, which we evaluated numerically for our eleven designs (Q3). As in earlier NTK analyses such as Du et al. (2019), the theorem is a sufficient condition for convergence. It is not meant to prescribe the width or the step size, and in our experiments the loss decreases without satisfying the condition (see Q3).

W2. Checkability for weak and nonlocal rows. For every loss trained in practice the certificate is a terminating decision procedure, for weak and nonlocal rows as well as pointwise ones. In a trained PINN the integrals of weak and nonlocal rows are evaluated by a fixed quadrature rule, so every row is a combination of finitely many point evaluations. For such rows positivity is equivalent to full row rank of the finite coefficient matrix B\mathbf B (Remark C.16, Sections D.3.2 and D.3.3), as it is a rank test per site for pointwise rows (Proposition 3.1, Section D.3.1). Both tests terminate and decide positive and singular designs. Every weak and nonlocal design in our experiments is of this kind. The remark on the truncation degree concerns only continuum functionals with exact integrals, which do not occur in training. There an independent design is certified at some finite moment degree (Proposition C.13, Corollary C.14), and a dependent design by an exact linear relation among the rows (Section D.3.3).

Q1. Yes. For weak and nonlocal rows evaluated by quadrature the finite rank test terminates and decides both positive and singular designs, as for pointwise rows (see W2).

Q2. Nonlinear constraints. Linear PDEs are the main restriction of the paper, and we say so. A nonlinear row is a nonlinear function of finitely many linear observations of the network, and its tangent Gram matrix factors exactly through their kernel at every width (Lemma F.3). The rank certificate still applies to that linear kernel. The deterministic limit fails. The factor multiplying the linear kernel depends on the network values at initialization, so the limiting kernel is random (Proposition F.5) and its smallest eigenvalue has no deterministic positive floor (Remark F.11). Theorem 4.2 transfers a deterministic gap to finite width, so this step has nothing to transfer. Whether a random but positive kernel still gives convergence is open (Remark F.18).

Q3. Size of the width condition. For each of the eleven designs we evaluated the limiting kernel by quadrature and found λmin⁡(K(0)∞)\lambda_{\min}(\mathbf K_{(0)}^\infty) between 1.6×10−71.6\times10^{-7} and 5.2×10−35.2\times10^{-3}. With λ0=λmin⁡(K(0)∞)\lambda_0=\lambda_{\min}(\mathbf K_{(0)}^\infty), the most favorable admissible choice, we evaluated the three terms of Equation (9) at δ=0.05\delta=0.05. The required width lies between 9.3×10169.3\times10^{16} and 4.5×10314.5\times10^{31}, and the term in λ0−3\lambda_0^{-3} dominates in every design. The table lists each design, with NN the number of rows.

DesignNNλmin⁡(K(0)∞)\lambda_{\min}(\mathbf K_{(0)}^\infty)Required width
Poisson 1D72.5×10−42.5\times10^{-4}1.3×10181.3\times10^{18}
Variable elliptic 2D135.2×10−35.2\times10^{-3}9.3×10169.3\times10^{16}
Heat 1D171.8×10−61.8\times10^{-6}1.8×10241.8\times10^{24}
Transport 1D156.2×10−76.2\times10^{-7}5.9×10255.9\times10^{25}
Wave 1D221.6×10−71.6\times10^{-7}1.4×10281.4\times10^{28}
Biharmonic 2D213.4×10−63.4\times10^{-6}4.5×10314.5\times10^{31}
Stokes 2D321.4×10−41.4\times10^{-4}1.6×10191.6\times10^{19}
Weak Poisson 1D64.5×10−54.5\times10^{-5}2.1×10182.1\times10^{18}
Nonlocal diffusion 1D59.9×10−69.9\times10^{-6}1.2×10201.2\times10^{20}
Integro-differential 1D63.0×10−53.0\times10^{-5}4.8×10194.8\times10^{19}
Caputo diffusion 1D171.6×10−61.6\times10^{-6}1.6×10241.6\times10^{24}

The largest width we trained is 16,38416{,}384. The experiments are therefore far outside the regime of Equation (9), which is why we wrote that they do not verify its constants. The condition is sufficient and far from necessary. At widths more than twelve orders of magnitude below it, the runs still show what the theorem concludes. The loss decreases, the kernel drift from initialization shrinks as the width grows, and the smallest eigenvalue of the empirical kernel stays positive. The constants RKR_K, B∗B_* and L∗L_* are evaluated on sample points, so the listed widths are lower bounds. The same list is in the README of the anonymous repository.

Reviewer_3oAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies positive-definiteness criteria for the limiting Neural Tangent Kernel (NTK) of scalar- or vector-valued physics-informed neural networks (PINNs) under linear constraints (including arbitrary finite-order strong-form differential rows, weak-form variational functionals, and finite-measure nonlocal observations), and provides finite-width gradient descent (GD) convergence guarantees with high probability. Theoretically, the proofs use the Matrix Chernoff inequality and Jacobian perturbations. Experimentally, 11 fixed linear PDE designs, 7 widths, and 10 random seeds are used for 3,000 steps of full-batch GD training, together with control experiments (mixed homogeneity analysis, three-row auditing, etc.) to test the theoretical bounds.

Critical strengths

It breaks the reliance of traditional NTK analysis on the "non-degeneracy assumption". Previous PINN-NTK literature (e.g., Wang et al., 2022) mostly assumes that the limiting Gram matrix is strictly positive definite, but lacks a means to test this assumption in concrete problems. This paper transforms the positive definiteness of the kernel into a directly computable matrix rank condition, filling the gap between theoretical analysis and practical application.

Critical weaknesses
  1. Although the paper explores the possibility of extending to deep networks (Depth-H) in Appendix F, its core rigorous proofs and quantitative convergence theorems are strictly confined to the "ordinary two-layer PINN." At a time when deep neural networks are increasingly becoming mainstream, this architectural limitation weakens its practical guiding significance.

  2. The paper points out that for genuinely nonlinear constraints (such as the convection term u⋅uxu\cdot u_x, the kernel converges to a random matrix rather than a deterministic matrix in the infinite-width limit, causing the current proof architecture to fail. This means that the method cannot be directly generalized to standard strong-form training for nonlinear PDEs, limiting its application to practical physical problems such as complex nonlinear flows.

  3. The paper is too long for a conference paper to read. It is more suitable for a journal with very careful checking.

Questions for authors
  1. In practical applications, for problems on high-dimensional or complex geometric domains, is it computationally easy for the construction and verification of the required matrices to suffer from the curse of dimensionality?

  2. For compatible singular designs, the paper handles singular kernels through active-subspace compression. How can this compressed subspace be automatically identified and constructed in actual training?

  3. How it is compared to directly computation positive definite of the finite-width Gram matrix? What's the distance to the limiting Gram matrix?

  4. Compared to Jin & Wu (2026a,b)、Gao et al. (2023), please try to give an example which they failed in NTK positivity, and you succeed in it and improves the training by your active-subspace compression/collocation redistribution, etc.

4 comments

Thank you for the review and the questions.

W1. Two-layer networks. Our aim was to show that an ordinary two-layer PINN already reaches zero training loss under plain gradient descent. A deeper network is at least as expressive, so two layers is the basic case. PINNs are in practice plain fully connected networks with a smooth activation (Raissi et al., 2019), so the relevant extension is depth (Appendix F). Lemma F.20 shows that at every depth one positive definite layer block certifies the kernel. Proposition F.22 carries Theorem 4.2 over to depth HH once a depth analogue of the row regularity holds. We also ran a new experiment with H=1,2,3H=1,2,3 hidden layers, widths 16 to 1,024 and five seeds on the Poisson 1D and heat 1D designs. The training loss decreases in all 120 runs and λmin⁡(K(k))\lambda_{\min}(\mathbf K_{(k)}) stays above half its initial value at every saved checkpoint. The table is in our response to Reviewer bx6a (Q3).

W2. Nonlinear constraints. Yes, this is the main restriction, and we say so and prove why the present argument fails. For a nonlinear PDE row the residual is a nonlinear function of the network's linear observations, so the loss is nonconvex in function space and the kernel is random already at initialization (Lemma F.3, Proposition F.5). Still, linear PDEs are not a niche. They include the heat, wave, Helmholtz, Maxwell, Stokes and Darcy equations. Nonlinear problems can also be approached through linear ones, by linearization or Picard iteration, and our theorem then applies to each linear step.

W3. Length. The proofs take about 32 of the 82 appendix pages, and they follow one structure for all three constraint classes.

  • Kernel. The limiting kernel, its finite-width gap and its stability near initialization are in Appendix B (pages 16 to 24, Definition B.1, Theorem B.9, Propositions B.10 and B.13).
  • Certificate. The results for pointwise, weak and nonlocal rows are in Sections C.2 to C.7 (pages 29 to 41, Theorems C.7, C.17 and C.21) and Appendix G (pages 71 to 77).
  • Gradient descent. Theorem 4.2 is proved once in Appendix H (pages 77 to 83, Theorem H.2). The three classes enter only through the constants of Definition B.1 and λ0\lambda_0.

The rest is experimental detail (Appendix K, pages 85 to 96), comparison with earlier theorems (Section C.1, pages 24 to 29) and worked examples and scope (Appendices D to F, I and J), which is the kind of material Q4 asks for.

Q1. High dimension and complex geometry. No, the certificate does not suffer from the curse of dimensionality. It uses no mesh, does not depend on the shape of the domain. For pointwise rows it is a rank check on one small coefficient matrix per collocation site, with one row per constraint at that site and one column per derivative term of the operator (Section D.3.1). The input dimension d0d_0 enters only through the number of these terms, for example d0d_0 for a Laplacian, and the cost is linear in the number of sites. For weak and nonlocal rows evaluated by quadrature the matrix has one row per constraint and one column per quadrature node (Sections D.3.2 and D.3.3).

Q2. Identifying the active subspace. Before training, the linearly dependent rows are identified as the left null vectors of the certificate matrices. Then one checks whether the target vector is orthogonal to these null vectors. If it is, the target is compatible and nothing needs to be done. Gradient descent runs on the original loss and converges to zero, and the compression enters only the analysis (Corollary E.4). If it is not, the target is incompatible and zero training loss is impossible for any optimizer (Lemma E.3).

Q3. Finite-width Gram matrix. The finite-width Gram matrix describes one sampled network at one iterate, and our experiments do compute its smallest eigenvalue at every saved checkpoint (Figure 1(a), Table 9). A theorem needs a bound that holds with high probability over the initialization and along the whole trajectory, which the limiting kernel provides. The distance to the limit is controlled in the proof by Propositions B.10 and B.13. In the experiments the relative operator-norm distance, as a median over the 110 runs at each width, falls from 0.220.22 at d1=16d_1=16 to 0.0550.055 at 256256, 0.0160.016 at 4,0964{,}096 and 0.00810.0081 at 16,38416{,}384 (Table 9). The fitted slopes are −0.512-0.512 to −0.469-0.469 (Section K.7), in line with the d1−1/2d_1^{-1/2} rate of the proof.

Q4. Examples. They are in Section C.1 with Table 2. Gao et al. (2023) use a cubic ReLU, for which a fourth-order row vanishes identically (Example C.1). Jin & Wu (2026a,b) assume distinct or separated sites, while Examples C.2 to C.4 place several rows at one site. Our rank criterion decides those cases, and Corollary E.4 gives convergence for duplicate rows, which Section K.5 confirms in training. These designs lie outside the hypotheses of the cited papers and are not counterexamples to them.

Thanks for the response. For Q4, Gao et al. (2023)'s cubic relu activation is relaxed to smooth activations in Xu et al. (2026, iclr), so it's not a principle difference. With much work on PINN NTK, this work gives a way to test the positivity of Gram matrix, but it does not bring foundmentally new insights into PINN NTK research. Besides, the Appendices are too long, full of colloraries, examples, with defensive description. So I keep the score and hope the paper can be human-friendly organized.

Thank you for the follow-up. Q4. We agree that Xu et al. (2026) remove the cubic ReLU restriction, so the activation is not the difference. However, there are several critical differences, and the row for Xu et al. (2026) in Table 2 (Section C.1) lists them.

  • PDE. Theirs is the heat equation of Gao et al. (2023). Ours is any fixed finite-order linear operator (Theorem C.7).
  • Constraint form. Theirs is pointwise residual and data rows. Ours also covers weak-form rows (Theorem C.17) and nonlocal rows (Theorem C.21).
  • Output. Theirs is scalar. Ours also covers coupled systems such as Stokes and linear elasticity.
  • Training points. Theirs requires that no two points are parallel, which excludes two rows at the same point. Ours allows several rows at one point and decides their independence by a rank condition (Theorem C.7).
  • Singular case. Theirs has no result when the Gram matrix is singular, and natural gradient descent needs its inverse. Ours identifies the nullspace and gives convergence on the active subspace for compatible targets (Corollary E.4).
  • Training. Their smooth-activation theorem is for natural gradient descent with only the hidden weights trained. Theorem 4.2 is for plain gradient descent with all parameters trained.

These cases are not niche. Table 1 lists the equations covered, including Poisson and variable-coefficient elliptic, transport, heat, wave, biharmonic, Stokes and linear elasticity, variational formulations with discontinuous coefficients, and integral and nonlocal-diffusion rows.

After the huge generalization of PDE cases and realistic training settings including no assumption that the collocation points are linear dependent, the insight of our paper is that whether training will converge can be checked before training starts, through our novel positivity check, as we show theoretically and confirm empirically.. The positivity check uses only the constraint rows and the training points. If it passes, Theorem 4.2 gives convergence of gradient descent to zero training loss at sufficient width.

Appendix. We agree and are open to shorten it. Our plan is to keep the NTK analysis (existence of the limiting kernel, the positivity certificate with one example per constraint class, the gradient descent proof) and the experimental protocol. We plan to remove the other examples and the scope material, including the proposition on depth(but adding the experiments as required by reviewers), and to reduce the comparison with earlier theorems to a short summary. We have been working on this and will upload the revision during the discussion period. If specific passages read as defensive or hard to follow, please name them and we will revise or remove them.

We have uploaded the revised version. The appendix is 44 pages instead of 81 and is organized along the argument: the common assumption and a roadmap of the proofs (Section B, Table 2), kernel existence (Section C), the nullspace theorems with each theorem followed by its proof (Section D), the certificates with one worked example per class (Section E), the gradient descent proof (Section F), and the experiments (Section G). The comparison examples, the sharpness constructions, the alternative criteria, and the nonlinear and depth material are removed. The general comment above lists every change. If specific passages still read as unclear, please name them and we will revise them.

Reviewer_bx6aOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when PINNs can reliably minimize their training loss. For two-layer networks with suitable activations and initialization, it proves that the limiting neural tangent kernel is positive definite exactly when the sampled linear constraints are independent. It gives rank-based criteria covering several examples. Using this positivity, the authors derive explicit network-width and step-size conditions guaranteeing geometric convergence of GD with high probability. They also show how redundant constraints can still permit convergence when the targets satisfy the same dependencies.

Critical strengths
  1. Main strength of the paper is that it provides conditions for when the limiting NTK can be singular, using the structural designconstraints.
  2. The paper goes beyond identifying an infinite-width kernel: it gives explicit sufficient conditions for discrete GD to converge at finite width. .
Critical weaknesses
  1. The experiments do not establish the practical usefulness of the sufficient bounds. They illustrate rank, conditioning, and kernel stability but do not verify the theorem’s width and step-size conditions.
  2. Some more discussion on the analytical and numerical results would be helpful; some paragraphs in the paper seem detached. For example, line (357-366)
Questions for authors
  1. How do the required network width and step size change as the number of constraints NN and the dimensions d0,d2d_0,d_2 increase? In particular, how do λ0\lambda_0 and Λ\Lambda scale with these parameters in Theorem 4.2?

  2. For one PDE experiment, could you calculate the width and step size required by Theorem 4.2 and compare them with those actually used?

  3. Do deeper networks show similar kernel stability and training behavior? Could you include a small experiment varying depth?

  4. In Figure 1(a), how should each error decrease as width increases? How does this depend on the number of constraints and the problem dimension?

  5. In Figure 1(b), why do different problems have different final losses despite similar condition numbers? Could you explain how the row-mixing parameter ε\varepsilon affects this behavior?

4 comments

Thank you for the review.

W1. Practical use of the sufficient bounds. Theorem 4.2 is a sufficient condition. Like earlier NTK convergence results (Du et al., arXiv:1810.02054), it proves convergence to zero training loss at sufficient width and is not meant to prescribe the width or step size. The experiments check the mechanism of the proof (a stable Gram matrix with smallest eigenvalue away from zero). As computed in Q2, the trained widths are far below the sufficient ones and the loss still decreases.

W2. Analytical and numerical discussion of the results, and lines 357 to 366. The appendix already contains analytical and numerical worked examples. Examples D.9 to D.11 carry out the certificate by hand for pointwise, weak and nonlocal rows, Example J.1 for a coupled advection-diffusion system, Corollaries C.19 and C.20 treat weak Poisson problems with a discontinuous coefficient, and Section K discusses each experiment. We agree that lines 357 to 366 read as a list of references. We will replace them by two sentences saying which hypotheses of Theorem 4.2 hold automatically and which one, the independence of the rows, has to be checked. Is this the disconnect you meant?

Q1. Dependence on NN, d0d_0 and d2d_2. In Eq. (9), NN appears only as log⁡(2N/δ)\log(2N/\delta), and d0d_0, d2d_2 do not appear. The rest enters through RKR_K, B∗B_*, L∗L_*, r‾δ\overline r_\delta and λ0\lambda_0. With a fixed number of loss groups and Ng−1/2N_g^{-1/2} scaling, the first four do not depend on NN (Remark B.12). Then η≤1/(2B∗2)\eta\leq1/(2B_*^2) does not change with NN, and the width changes only through log⁡N\log N and λ0\lambda_0. The dimensions enter through the per-neuron bounds BiB_i, LiL_i and λ0\lambda_0, for which we have no general rate. The width grows like λ0−3\lambda_0^{-3}.

Q2. Required versus used width and step size. For Poisson 1D, λmin⁡(K(0)∞)=2.46×10−4\lambda_{\min}(\mathbf K_{(0)}^\infty)=2.46\times10^{-4}. With λ0\lambda_0 equal to this value, the most favorable choice, Eq. (9) at δ=0.05\delta=0.05 requires d1≥1.25×1018d_1\geq1.25\times10^{18} and η≤0.0496\eta\leq0.0496. We trained widths up to 16,38416{,}384 with η=0.2/λmax⁡(K(0))≈0.48\eta=0.2/\lambda_{\max}(\mathbf K_{(0)})\approx0.48. Over all eleven designs the required width lies between 9.3×10169.3\times10^{16} and 4.5×10314.5\times10^{31}, dominated by the λ0−3\lambda_0^{-3} term. The computation is in the repository (width_requirement.py https://github.com/pinn-ntk-certificates/iclr2027/blob/main/experiments/theorem_audit/width_requirement.py). In particular, the largest trained width is more than twelve orders of magnitude below the required one in every design, and the loss still decreases in every run.

Q3. Deeper networks. By Lemma F.20, at every depth the Gram matrix is a sum of positive semidefinite layer blocks, so one positive definite block certifies the kernel. Proposition F.22 carries Theorem 4.2 over to depth HH under a depth analogue of row regularity, which is open (Conjectures F.23 and F.25). We ran the suggested experiment with H=1,2,3H=1,2,3, widths 16 to 1,024, five seeds, the Poisson 1D and heat 1D designs and the settings of Section 5. (depth_variation.py).

DesignHHλmin⁡(K(0))\lambda_{\min}(\mathbf K_{(0)})Kernel drift, dd = 16 / 64 / 256 / 1,024LT/L0\mathcal L_T/\mathcal L_0
Poisson12.61e-44.54 / 1.76 / 0.60 / 0.191.15e-8
Poisson21.08e-44.67 / 1.80 / 0.56 / 0.181.08e-8
Poisson32.61e-55.33 / 2.18 / 0.69 / 0.239.76e-8
Heat11.73e-60.60 / 0.17 / 0.042 / 0.0121.32e-4
Heat25.85e-70.74 / 0.39 / 0.13 / 0.0351.63e-3
Heat31.26e-72.21 / 0.87 / 0.35 / 0.107.96e-3

Entries are medians over five seeds, with λmin⁡\lambda_{\min} and the loss ratio at d=1,024d=1{,}024. The drift is the largest relative change of the Gram matrix over saved checkpoints. At all depths it decreases with width, and in all 120 runs the loss decreases and λmin⁡(K(k))\lambda_{\min}(\mathbf K_{(k)}) stays above half its initial value. Depth does not lower the loss after 3,000 steps. λmin⁡(K(0))\lambda_{\min}(\mathbf K_{(0)}) decreases with depth, consistent with slower convergence at the same step-size rule.

AuthorsOpenReview ↗

continued.

Q4. Figure 1(a). The proof bounds all three errors by order d1−1/2d_1^{-1/2}. The initial kernel error is an average of d1d_1 independent terms (Proposition B.10), and each neuron moves by order d1−1/2d_1^{-1/2}, which bounds the drift (Proposition B.13) and the frozen-kernel discrepancy. The fitted slopes are −0.512-0.512 to −0.469-0.469 for the initial error (Section K.7) and −0.97-0.97 and −0.88-0.88 for the drift and the frozen discrepancy between d1=256d_1=256 and 16,38416{,}384 (Table 9). NN and the dimension change only the prefactors (Q1).

Q5. Figure 1(b) and the row-mixing parameter. ε\varepsilon controls the condition number without changing the rank. The second of two independent rows ℓ1,ℓ2\ell_1,\ell_2 is replaced by (ℓ1+εℓ2)/1+ε2(\ell_1+\varepsilon\ell_2)/\sqrt{1+\varepsilon^2}. For ε>0\varepsilon>0 the kernel stays positive definite while λmin⁡/λmax⁡\lambda_{\min}/\lambda_{\max} decreases like ε2/2\varepsilon^2/2 (Section K.6). The loss after a fixed number of steps also depends on how the initial residual is split between the two eigendirections. With η=0.2/λmax⁡\eta=0.2/\lambda_{\max} the strong direction is removed in a few steps, and at ε=0.01\varepsilon=0.01 the weak one loses less than 8%8\% in 3,0003{,}000 steps. The final loss ratio is therefore the share of the initial residual in the weak direction, about 0.190.19, 0.400.40 and 0.850.85 for the pointwise Poisson, weak boundary-residual and Stokes controls, which reproduces the ratios 0.170.17, 0.370.37 and 0.800.80 of Table 8.

Thank you for the clarifications and additional experiments. Please include the depth experiments and the discussion of the limitations of the theoretical width bounds in the revised paper. Your response addresses most of my concerns, and I maintain my score.

Thank you. We will add the depth experiment to the appendix, and the comparison of required and trained widths from Q2 to the discussion of Theorem 4.2, highlighting it is a sufficient but not necessary requirement.

Reviewer_6BQtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper derives a simple, computable condition of positive definiteness when training a PINN with strong-form, weak-form, and integral constraints. This positive-definite-ness is important in NTK theory, since it assures the global convergence of sufficiently wide neural networks for sufficiently small learning rate. For linear PDEs, the authors give two characterisations: the regularity conditions for the convergence of the initial NTK to the limiting, deterministic NTK and the linear-independence condition for the positive-definiteness of NTK. Under the assumption that the network is a two-layer NN, the authors also prove the quantitative convergence, certifying that their claim is sufficient to prove the global convergence of PINN trained with GD. The authors further empirically verify their claim by showing that (1) wider networks show better kernel convergence behaviour and smaller final loss, and (2) when the NTK is ill-conditioned, the network fails to converge.

Critical strengths

The global convergence of the PINN is well-studied, however, they are largely limited to specific PDEs. Also, most of the NTK theories in PINN assumes the positive definiteness of the NTK as assumption, as it is in general numerically computable. While this is true, computing the NTK and its smallest eigenvalue for complicated PDEs are often computationally challenging, and might require solving large matrix system, which is often intractable. So I believe simple checkable condition for the global convergence is definitely interesting problem, and allows the PINN researchers to not believe, but being assured for their NTK intuition to hold.

Critical weaknesses

The largest weakness in this paper is the applicability of the theory.

First, the theory assumes the PDE in question to be linear. While there are interesting linear PDEs, most of the practically appearing PDEs are nonlinear, for instance, Navier-Stokes, Hamilton-Jacobi-Bellman, KdV, etc. In PDEBench (https://arxiv.org/abs/2210.07182), among 8 PDEs used as benchmarks, only two of them (Advection, Darcy flow) are linear PDEs. So this applicability is quite concerning, in terms of the problems that can be tackled.

On page 9, the authors discuss this problem: the NTK Gram matrix does not converge to a deterministic limit (It also diverges as training continues, in general); therefore, the extension of NTK theory to nonlinear problems is open. This is true for vanilla NTK theory even for usual cross-entropy loss (for instance, see https://arxiv.org/abs/2504.11130). On the other hand, there are also applications of NTK theory to classification problems, which do not assume indefinite training (https://arxiv.org/abs/1909.12292v4), so extending this theory to obtain NTK-type convergence for nonlinear PDE may not be impossible.

Another problem is the fixed design, where authors assume that the collocation points are fixed before training. This is rare in PINN training, and authors rely on the stability theory to transfer the global convergence at these finite points to the solution loss. Again, this is a problem in the vanilla NTK theory. One can show that when the input domain is bounded, infinitely many training points give the smallest eigenvalue of the NTK to converge to 0, making the positive-definiteness-based proof impossible even in standard training. One hope we can have is that authors not just prove the positive-definiteness, but characterises the kernel of limiting kernel. If one incorporates the target function into the convergence analysis, one can consider the restricted eigenspace, or Rayleigh quotient, as in (https://proceedings.neurips.cc/paper_files/paper/2022/file/45d74e190008c7bff2845ffc8e3facd3-Paper-Conference.pdf). So extending the current theory to full-batch GD might be possible.

Overall, the authors' contribution is largely limited by the limitations of the vanilla NTK theory. As I suggested, it doesn't mean that we can't obtain global convergence on nonlinear PDEs or in the random design case, even under the NTK theory. So, an extension to such cases would be needed to resolve this limited applicability.

Questions for authors
  • I don't think extension of full theory to either nonlinear PDE or variable design is possible within rebuttal period. One way is considering specific example. For instance, on two-layer NN (possibly not training second layer), can one use the theories I mentioned to prove convergence in nonlinear PDE (it would be okay to be specific one) or variable design?
  • Most of the proof idea is incomprehensible. I appreciate that the authors gives explicit pointer to each claim (as the paper is very long), it is not very easy to follow. Can you give proof technique instead?
1 comment

Thank you for the careful review and the pointers.

W1. Only linear PDEs. Yes, this is the main restriction, and we say so. Still, linear PDEs are not a niche. They include the heat, wave, Helmholtz, Maxwell, Schrödinger, linear elasticity, Stokes and Darcy equations, and in finance everything priced through Feynman–Kac, where nonlinearity enters only with control or free boundaries. Nonlinear problems can also be approached through linear ones, by linearization or Picard iteration, and our theorem then applies to each linear step.

W2. Nonlinear rows. Agreed that a random kernel limit is not fatal by itself. From a quick look at the papers you point to (Yu, Tian & Chen, arXiv:2504.11130; Ji & Telgarsky, arXiv:1909.12292), the mechanism there seems different. The logits enter the loss linearly and the loss is convex in them, so the kernel only drifts late in training because cross-entropy has no minimizer. For a nonlinear PDE row the residual is a nonlinear function of the network's linear observations, so the loss is nonconvex in function space and the kernel is random already at initialization (Lemma F.3, Prop. F.5). Whether a random but uniformly positive kernel still gives convergence is open, as Remark F.18 says. The page-9 phrase "negative rather than open" refers only to the deterministic population kernel, and we will reword it.

W3. Fixed design. Definition 2.4 is already full-batch GD on the fixed design, so we may have misread the suggestion. A forward PINN loss has no noise, so conditional on the sampled points a random design is a fixed design. On passing from convergence at the collocation points to the solution error, we rely on the stability estimates recalled in Section 6, which bound the solution error by a stability constant times the training residual plus a quadrature term of the design. Our theorem controls the first term. The quadrature term is fixed by the design and is the same for any optimizer.

Q1. We think the first concern does not apply and the second is not applicable in the form asked. For the design, the setting you describe, all collocation points fixed and used in every update, is Definition 2.4, and Theorem 4.2 is proved for it. For a nonlinear PDE, the NTK route needs a deterministic gap to transfer, and W2 shows there is no deterministic one already at initialization, whether or not the second layer is trained.

Q2. GD Convergence Proof technique. Three steps. (1) Kernel. A two-layer network is a sum over neurons and every linear row commutes with the sum, so each row is a normalized sum of i.i.d. per-neuron terms (Definition B.1). The law of large numbers gives the limiting kernel, matrix Chernoff its finite-width gap, and a per-neuron Lipschitz bound its stability near initialization (Appendix B). The input is the three bounds of Definition B.1, which hold for pointwise rows by smoothness of σ\sigma and for weak or nonlocal rows because they are bounded functionals of the one-neuron function. (2) Certificate. The nullspace of the output block is the set of linear relations among the feature functions ξ↦fi(ξ)\boldsymbol\xi\mapsto\mathbf f_i(\boldsymbol\xi). Analyticity makes a relation on the support a relation everywhere, and classwise arguments (independence across sites for pointwise rows, density of ridge functions for weak and nonlocal rows) reduce it to exact relations among the rows, which is a finite rank check (Section 3, Appendices C and G). (3) Gradient Descent convergence. Following similar structure in Du et al. (arXiv:1810.02054), strong induction on (94) shows that the iterates stay in a per-neuron tube and the loss decays geometrically. Lemmas H.5–H.6 bound the displacement, Prop. H.4 keeps the gap, and Prop. H.3 with Lemmas H.7–H.8 gives the one-step contraction. Nothing in Appendix H depends on the constraint class except through the constants of Definition B.1 and λ0\lambda_0, which is what unifies the three classes. We will add this overview into the maintext.

Comments on the paper

7 comments

Thank you for the follow-up and for raising your score. We are glad the linearity point is settled, and we will add the Picard-iteration remark to Section 6.

Random design per step. Thank you for the clarification and the three references (arXiv:2305.19685, arXiv:2206.02016, arXiv:2404.07662); we had read the comment differently. We agree that adaptive, network-dependent sampling is a strength of PINNs that our fixed-design theorem does not cover. On a fixed design the PINN loss is a noiseless interpolation problem, so the fixed design itself is not what limits the result, but with many collocation points full-batch training is expensive, and per-step resampling is the practical answer. We will say this in Section 6, cite the three papers, and point to Robin, Scaman & Lelarge (arXiv:2301.08117).

Length. The proofs take about 32 of the 82 appendix pages, and they follow one structure for all three constraint classes.

  • Kernel. The limiting kernel, its finite-width gap and its stability near initialization are in Appendix B (pages 16 to 24, Definition B.1, Theorem B.9, Propositions B.10 and B.13).
  • Certificate. The results for pointwise, weak and nonlocal rows are in Sections C.2 to C.7 (pages 29 to 41, Lemma C.11, Theorems C.17 and C.21) and Appendix G (pages 71 to 77, Proposition G.2).
  • Gradient descent. Theorem 4.2 is proved once in Appendix H (pages 77 to 83, Theorem H.2). The three classes enter only through the constants of Definition B.1 and λ0\lambda_0.

The rest is experimental detail (Appendix K, pages 85 to 96), comparison with earlier theorems (Section C.1, pages 24 to 29) and worked examples and scope (Appendices D to F, I and J).

GD Convergence Proof Technique. Two main steps differ from Du et al. (arXiv:1810.02054). The entrywise Hoeffding bound with a union over N2N^2 entries is replaced by matrix Chernoff on the per-neuron Gram terms, which lowers the width from N2λ0−2log⁡NN^2\lambda_0^{-2}\log N to λ0−1log⁡N\lambda_0^{-1}\log N (Prop. B.10, Remark B.12). The ReLU activation-pattern argument for the kernel near initialization is replaced by a deterministic Lipschitz perturbation of the Jacobian, which smoothness of the activation allows (Prop. B.13, Prop. H.4).

Thank you for the review.

Main contribution. We see the main contribution differently. For us it is the pair formed by the checkable certificate and the gradient descent theorem. The certificate decides whether the limiting kernel of a given design is positive definite, and Theorem 4.2 then gives one finite-width gradient descent argument, with all parameters trained, for strong-form, weak-form and nonlocal rows. The nullspace formulas are one step of the certificate.

W1. Quantitative content of Theorem 4.2. The quantitative content of the theorem is its explicit width and step-size rule, Equation (9), and we have now evaluated that rule for all eleven designs. We agree that the experiments do not show the rule to be useful as a prescription or close to necessary, and we do not claim that it is. What they test is a different question, whether the mechanism of the proof, a Gram gap that survives training, is present at the widths one can train.

Theory. Theorem 4.2 is a sufficient condition, as the earlier NTK convergence results are (Du et al., arXiv:1810.02054). We do not claim that the λ0−3\lambda_0^{-3} width is sharp or that Proposition E.5 closes the gap, and the main text makes no quantitative claim about the smallest eigenvalue of the limiting kernel in Theorem 4.2. This is because a lower bound on λ0\lambda_0 has to be evaluated design by design, so we do not pursue one. However, what we prove instead is a checkable procedure that decides, for a given design, whether the limiting kernel is positive definite, which is the premise the theorem needs. In addition, to make the size of the condition explicit, we have now evaluated Equation (9) for the eleven designs of Section 5 with λ0=λmin⁡(K(0)∞)\lambda_0=\lambda_{\min}(\mathbf K_{(0)}^\infty), the most favorable admissible choice, and δ=0.05\delta=0.05. The smallest eigenvalue lies between 1.6×10−71.6\times10^{-7} and 5.2×10−35.2\times10^{-3}, the required width between 9.3×10169.3\times10^{16} and 4.5×10314.5\times10^{31}, and the λ0−3\lambda_0^{-3} term dominates in every design. For Poisson 1D the condition asks for d1≥1.25×1018d_1\geq1.25\times10^{18} and η≤0.0496\eta\leq0.0496. The computation is in the repository (experiments/theorem_audit/width_requirement.py), and the table of the eleven values goes into the experiments appendix as well as in Q3 for reviewer CLWs.

Experiments. They were never meant to verify Equation (9), and Section 5 says so. They check the mechanism of the proof, a Gram matrix whose smallest eigenvalue stays away from zero along training, at widths one can train. We trained widths up to 16,38416{,}384 with η=0.2/λmax⁡(K(0))≈0.48\eta=0.2/\lambda_{\max}(\mathbf K_{(0)})\approx0.48, more than twelve orders of magnitude below the sufficient width and about ten times the admissible step. There the empirical kernel keeps at least half of the reference gap at every checkpoint in all 110 runs at each tabulated width above 16, and the training loss decreases in every one of the 770 runs. The behavior the theorem and its proofs describes sets in long before its constants are met, so the constants are not a prescription and the sufficient condition is far from necessary.

W2. Length. We agree that the appendix can be too long. However, its proofs follow one structure for the three constraint classes and take only about 32 of the 82 appendix pages. The kernel limit, its finite-width gap and its stability near initialization are in Appendix B (pages 16 to 24). The nullspace results are in Sections C.2 to C.7 (pages 29 to 41), with the pointwise case completed in Appendix G (pages 71 to 77). Theorem 4.2 is proved once in Appendix H (pages 77 to 83), and the three classes enter only through the constants of Definition B.1 and λ0\lambda_0. The other 50 pages are comparison, examples and scope. Part of them is material that reviewers ask for, such as Appendix F on nonlinear constraints and depth, which answers questions of the other reviewers. We will consider shortening the appendix, in particular the comparison of scopes.

W2. Verification. We address the four verification points as follows. 1. Lemma C.8 gets a self-contained proof (Q1). 2.The derivation of Example C.10 was explicit in the appendix but now has more details. 3.The sentence "In the generated paragraph below" in Section K.5 is deleted. Section K documents the experimental protocols and diagnostics, and that sentence introduced the table of the three-row audit, so it concerns the description of an experiment and no theoretical result. 4.For the tracked artifacts, we reran the whole campaign of Section K, that is the 770 PDE training runs, the 1,080 two-row control runs and the 7,700 initialization kernels, from the tracked configuration and kept every per-run record (Q3).

Q1. Lemma C.8. The proof below is self-contained, no longer depends on the preprint. It turns a linear relation among the QrQ_r into one among polynomials in x\mathbf x.

Proof. Let ∑rcrQr≡0\sum_r c_rQ_r\equiv0 on Rd0+1\mathbb R^{d_0+1} and fix a multi-index β\beta. At w=0\mathbf w=0 one has ∂wγwα=α! δγα\partial_{\mathbf w}^{\gamma}\mathbf w^{\alpha}=\alpha!\ \delta_{\gamma\alpha} and, for α≤β\alpha\leq\beta, ∂wβ−ασ(∣α∣)(w⊤xr+v)=xrβ−ασ(∣β∣)(v)\partial_{\mathbf w}^{\beta-\alpha}\sigma^{(\vert\alpha\vert)}(\mathbf w^\top\mathbf x_r+v)=\mathbf x_r^{\beta-\alpha}\sigma^{(\vert\beta\vert)}(v), so the Leibniz rule gives

∂wβ[wασ(∣α∣)(w⊤xr+v)]∣w=0=(∂αmβ)(xr) σ(∣β∣)(v),mβ(x)=xβ.\partial_{\mathbf w}^{\beta}\big[\mathbf w^{\alpha}\sigma^{(\vert\alpha\vert)}(\mathbf w^\top\mathbf x_r+v)\big]\Big\rvert_{\mathbf w=0}=(\partial^{\alpha}m_\beta)(\mathbf x_r)\ \sigma^{(\vert\beta\vert)}(v),\qquad m_\beta(\mathbf x)=\mathbf x^{\beta}.

Hence σ(∣β∣)(v) Λ(mβ)=0\sigma^{(\vert\beta\vert)}(v)\ \Lambda(m_\beta)=0 for all vv, where Λ(f)=∑rcr∑α∈Jrκr,α ∂αf(xr)\Lambda(f)=\sum_r c_r\sum_{\alpha\in J_r}\kappa_{r,\alpha}\ \partial^{\alpha}f(\mathbf x_r). Since σ\sigma is not a polynomial, no σ(n)\sigma^{(n)} vanishes identically, so Λ\Lambda vanishes on every monomial, hence on all polynomials. Fix r0r_0, let KK be the largest order in ⋃rJr\bigcup_rJ_r, choose α0∈Jr0\alpha_0\in J_{r_0} of maximal order, and set

P(x)=(x−xr0)α0α0!∏r≠r0(∣x−xr∣2∣xr0−xr∣2)K+1.P(\mathbf x)=\frac{(\mathbf x-\mathbf x_{r_0})^{\alpha_0}}{\alpha_0!}\prod_{r\neq r_0}\left(\frac{\vert\mathbf x-\mathbf x_r\vert^2}{\vert\mathbf x_{r_0}-\mathbf x_r\vert^2}\right)^{K+1}.

Every derivative of PP of order at most KK vanishes at each xr\mathbf x_r with r≠r0r\neq r_0, and ∂αP(xr0)=δαα0\partial^{\alpha}P(\mathbf x_{r_0})=\delta_{\alpha\alpha_0} for ∣α∣≤∣α0∣\vert\alpha\vert\leq\vert\alpha_0\vert. Therefore 0=Λ(P)=cr0κr0,α00=\Lambda(P)=c_{r_0}\kappa_{r_0,\alpha_0}, so cr0=0c_{r_0}=0. □\square

Q2. Example C.10. The derivation is printed in full in the example. We add more details now. The setting is one row λ(f)=12∫−11f\lambda(f)=\tfrac12\int_{-1}^{1}f with vT=0v_T=0, so h(z)=aF(w,b)h(z)=aF(w,b) with F(w,b)=λ(tanh⁡(w ⋅+b))F(w,b)=\lambda(\tanh(w\ \cdot+b)), w∈[−0.1,0.1]w\in[-0.1,0.1], b∈[1,2]b\in[1,2], a∈[−0.1,0.1]a\in[-0.1,0.1], R=0.1R=0.1 and δ=0.05\delta=0.05.

  1. Gap. On the support wx+b≥0.9wx+b\geq0.9, so F≥tanh⁡(0.9)F\geq\tanh(0.9) and λ0=tanh⁡(0.9)2=0.5130826\lambda_0=\tanh(0.9)^2=0.5130826.
  2. First derivatives. ∣F∣≤1\vert F\vert\leq1, ∣Fw∣≤12\vert F_w\vert\leq\tfrac12 and ∣Fb∣≤1\vert F_b\vert\leq1. With ∣a∣≤0.1\vert a\vert\leq0.1 on the support, RK≤1+0.12⋅54=1.0125R_K\leq1+0.1^2\cdot\tfrac54=1.0125, and with ∣a∣≤0.2\vert a\vert\leq0.2 on the tube, Λ=B∗2=1+0.22⋅54=1.05\Lambda=B_*^2=1+0.2^2\cdot\tfrac54=1.05.
  3. Second derivatives. With c2=∥tanh⁡′′∥∞=4/(33)c_2=\Vert\tanh''\Vert_\infty=4/(3\sqrt3), ∣Fww∣≤c2/3\vert F_{ww}\vert\leq c_2/3, ∣Fwb∣≤c2/2\vert F_{wb}\vert\leq c_2/2 and ∣Fbb∣≤c2\vert F_{bb}\vert\leq c_2, and the Hessian of hh gives L∗2≤0.22c22(19+12+1)+2⋅54=30839/12150<2.5382L_*^2\leq0.2^2c_2^2(\tfrac19+\tfrac12+1)+2\cdot\tfrac54=30839/12150<2.5382.
  4. Initial residual. A0,∗=0.1A_{0,*}=0.1 and H0≤0.1/3H_0\leq0.1/\sqrt3, so r‾0.05≤0.1/3+0.12log⁡40=0.3293553\overline r_{0.05}\leq0.1/\sqrt3+0.1\sqrt{2\log40}=0.3293553.
  5. Width. With N=1N=1 and cCh=14+34log⁡34=0.0342384c_{\mathrm{Ch}}=\tfrac14+\tfrac34\log\tfrac34=0.0342384, the three terms of Equation (9) are 212.6122212.6122, 1355.97901355.9790 and 692.2517692.2517, so d1≥1356d_1\geq1356 and η≤1/(2Λ)=10/21\eta\leq1/(2\Lambda)=10/21.

The width is small because λ0≈0.51\lambda_0\approx0.51.

Q3. Numbers that cannot be regenerated. We have rerun the whole campaign (770 training runs, 1,080 control runs, 7,700 initialization kernels) from the tracked configuration with every per-run record kept, so every number of the main text and Tables 5 to 10 is now regenerated from tracked artifacts. The rerun reproduces every value of the submission (767 of 770, 17/110 and 110/110, the Table 9 medians, the slopes −0.512-0.512 to −0.469-0.469, every tally and count of Tables 7 and 8 and Sections K.5 and K.6), except one Table 8 loss ratio at the float64 floor (6.0×10−306.0\times10^{-30} for 6.36×10−306.36\times10^{-30}). The records (620 MB) are the assets of release campaign-records-20261115 of the repository, with a hash list and restore instructions in CAMPAIGN_STATUS.md.

Before the rerun, released commands regenerated only the three-row audit, Table 10 and the eleven certificates. The tracked aggregate stored Table 8, Figure 1(b), the GD-outcome counts of Table 7 and the Section K.5 counts without regenerating them. The remaining values could not be regenerated: Table 5, the Gram-status and certificate tallies of Table 7, Table 9 and Figure 1(a), and in Section K the counts 812, 612, 200 and 767, the factors 694 and 416, the "23 of 24" settings, the Poisson baseline and the slopes. For these values the statement on tracked artifacts did not hold, and we have corrected it. The per-run histories were not retained because the campaign ran as cluster job arrays and only aggregated summaries were collected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.