acceptodds
Under review as a conference paper at ICLR 2027

Unified generalization analysis for physics informed neural networks

Abstract

Physics-Informed Neural Networks (PINNs) and their variational counterparts (VPINNs) are neural networks that incorporate physical laws, making them useful for scientific problems. Existing generalization analyses for PINNs and VPINNs remain limited, often requiring restrictive assumptions such as stability conditions or linear ellipticity. In this paper, we derive generalization bounds for neural networks that involve differentiation with respect to input variables, covering PINNs and VPINNs under a unified framework. We apply Taylor expansion to represent nonlinear differential operators as linear operators on a high-dimensional space, enabling the use of Koopman-based analysis and showing that high-rank networks can generalize well even in settings involving differential operators. We also show that the nonlinearity of the differential operator exponentially enlarges the bound, highlighting its significant impact on generalization.

Primary area:
learning theory
Keywords:
generalization bound, physics informed neural network, Koopman operator

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fhaNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is to study a generalization error of PINNS.

Critical strengths

write down some mathematical analysis of PINNS

Critical weaknesses

The crucial point is that the solution error is too large even for low-dimensional equations.

Questions for authors

Numerical methods can achieve at a high accuracy, it is not clear how good for PINNS

1 comment

Thank you for your review.

On the weakness (solution error of PINNs) and the question (comparison with numerical methods)

We understand the concern to be about the accuracy of PINNs as PDE solvers compared with classical numerical methods. We would like to clarify the scope of the paper. We do not claim that PINNs or VPINNs are more accurate than classical solvers, and we do not propose a new solver. The contribution is theoretical: a generalization bound for neural networks whose loss involves derivatives with respect to the input, i.e. a bound on the gap between the residual loss on the collocation points and its expectation. The analysis covers PINNs and VPINNs, and also other models with differential operators, such as Hamiltonian and Lagrangian neural networks.

The accuracy of PINNs is not guaranteed in the way it is for classical methods. It is therefore important to understand which factors control their error. Our bounds identify several of them: the number of collocation points, the determinants of the weight matrices, the order rr of the nonlinearity of the differential operator, and the test function. For example, the bound diverges as the variational model approaches the standard PINN (Remarks 4.10 and 4.11), which suggests an advantage of the variational form in terms of generalization.

Your point also touches a real limitation, which we now state explicitly in Section 6 (marked in red in the revised PDF). Our bound concerns the generalization of the residual, not the error of the solution. A small residual implies a small solution error only if the PDE is stable, and bounding the solution error on the basis of our bound is left for future work.

On soundness

The review does not point to a specific statement or proof. If there is a particular step that you consider incorrect, we would be grateful if you could indicate it, and we will address it. In the revision, we corrected the issues of mathematical rigor raised by the other reviewers: the nonlinear lifting is now defined as a pointwise lift (Section 3, Assumption 3.2), the Koopman-based bounds are re-derived for it (Lemma 4.7, Theorem 4.9), the Monge–Ampère experiment is covered by an expansion at a valid point with an explicit remainder (Appendix D), and the gap between the determinant factor and the regularizer is discussed (Appendix C).

Reviewer_D8YbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a unified generalization analysis for PINNs and VPINNs. The main idea is to use Taylor expansion to represent nonlinear differential operators as linear operators in a higher-dimensional space, enabling the application of Koopman-based generalization bounds. The resulting bounds characterize the effects of network weights and operator nonlinearity on generalization. The authors further design a bound-inspired regularization term, evaluate it on the Navier–Stokes equation, and investigate the relationship between the regularization factor and test error on the Monge–Ampère equation.

Critical strengths

The paper provides a unified perspective on generalization bounds for PINNs and VPINNs, covering linear and certain nonlinear differential operators. I find the combination of operator Taylor expansion and Koopman-based analysis relatively novel and interesting. In particular, representing nonlinear differential operators through higher-dimensional linear operators offers a potentially useful way to extend existing network generalization theory.

Critical weaknesses
  1. The applicability of Taylor approximation to general nonlinear operators is unclear. The proposed framework relies on Taylor expansion, but it is not clear how accurately general nonlinear differential operators can be approximated under the theoretical assumptions. The Navier–Stokes convection term is already quadratic. Although the Monge–Ampère example involves a non-polynomial operator, the approximation error is not quantified. Table 1 reports different powers rr of the regularization factor, but does not directly examine the Taylor truncation error ϵ\epsilon or the trade-off discussed in Remark 4.4. More importantly, the theory uses an rr-order Taylor approximation of gg at zero, whereas the Monge–Ampère example involves log⁡det⁡D2u\log\det D^2u, which is not defined at a zero Hessian. It is therefore unclear how the stated theory applies to this example. A more challenging nonlinear example, together with an analysis of the approximation error, would strengthen the claimed generality.

  2. The experimental validation of the regularization term is not fully convincing. In Figure 1, the improvements appear relatively modest, and the curves follow similar trends. More importantly, the implementation uses a modified proxy rather than the exact factor appearing in the theoretical bound (Appendix C). Figure 2 relates this proxy to the test error on Monge–Ampère, but there is no evidence that the regularizer actually reduces the relevant factor on Navier–Stokes. Moreover, only one regularization coefficient (0.01) is considered. Tracking the factor with and without regularization, and examining the sensitivity to the coefficient, would make the experiments more convincing.

  3. There seems to be a mismatch between the theoretical assumptions and the experimental architecture. The bound contains the factor ∣det⁡(Wl∗Wl)∣−1/4|\det(W_l^*W_l)|^{-1/4}, but the experimental network uses an output matrix W4∈R3×64W_4\in\mathbb R^{3\times64}, for which det⁡(W4∗W4)=0\det(W_4^*W_4)=0. Although Appendix C uses nonzero singular values for numerical computation, it is unclear how this modification relates to the original theoretical bound. This discrepancy should be explicitly addressed.

  4. The connection to physical solution accuracy could be discussed further. The proposed bound concerns the generalization of PDE residuals, but a smaller residual loss does not necessarily imply a more accurate physical solution. While this is not a fundamental weakness of the theoretical contribution, it would be interesting to understand whether the bound can provide any guidance on the actual solution error, rather than only the residual generalization gap.

Questions for authors
  1. How does the theory apply to the Monge–Ampère example, where gg involves log⁡det⁡\log\det and is not defined at zero? Can the authors quantify the Taylor approximation error ϵ\epsilon for this case?

  2. Does the implemented regularizer actually reduce the relevant factor in the theoretical bound? How sensitive are the results to the regularization coefficient and the modifications introduced in Appendix C?

  3. How should the determinant term be interpreted when the weight matrix is not full column rank, as in the experimental output layer?

  4. Is there a possible connection between the proposed residual generalization bound and the error with respect to the true PDE solution?

2 comments

Thank you for the constructive comments. We revised the paper along each of your four points. All changes are marked in red in the revised PDF.

W1/Q1: Taylor approximation, Monge–Ampère, and the error ϵ\epsilon

Expansion point (Section 5.2, new Appendix D.1). You are right that g(a,H)=−a+log⁡det⁡Hg(a,H)=-a+\log\det H is not defined at 00. Proposition D.1 extends Theorem 4.6 to an expansion at any point a0a_0 of an open domain OO of gg, with wθ=D~uθ−a0w_\theta=\tilde{\mathcal D}u_\theta-a_0 in place of D~uθ\tilde{\mathcal D}u_\theta. We take a0=(0,A0)a_0=(0,A_0) with A0=D2u∗=diag(2,1)A_0=D^2u^\ast=\mathrm{diag}(2,1) and O={(a,H):H positive definite}O=\lbrace(a,H): H\text{ positive definite}\rbrace.

Quantification of ϵ\epsilon (Appendix D.2). Let ρθ\rho_\theta be the supremum over X\mathcal X of the spectral radius of A0−1(D2uθ−A0)A_0^{-1}(D^2u_\theta-A_0). If ρθ<1\rho_\theta<1, Proposition D.5 gives the remainder RrR_r in closed form and the bounds ∣Rr(H)∣≤mρr+1/((r+1)(1−ρ))|R_r(H)|\le m\rho^{r+1}/((r+1)(1-\rho)) and ∣Rr(H)∣≤ϵr∣w∣r|R_r(H)|\le\epsilon_r|w|^r with ϵr=crρθ∥A0−1∥opr/((r+1)(1−ρθ))\epsilon_r=c_r\rho_\theta\Vert A_0^{-1}\Vert_{\mathrm{op}}^r/((r+1)(1-\rho_\theta)), where c1=mc_1=\sqrt m and cr=1c_r=1 for r≥2r\ge2. The truncation error thus decays geometrically in rr.

Trade-off of Remark 4.4 (Table 1). Table 1 now reports ϵr\epsilon_r, the measured truncation error τr=∥Rr(Hθ)∥L2(X)\tau_r=\Vert R_r(H_\theta)\Vert_{L^2(\mathcal X)}, and the main term (∣X∣+∑i≤r∥wθ∥L2i2i)1/2(|\mathcal X|+\sum_{i\le r}\Vert w_\theta\Vert_{L^{2i}}^{2i})^{1/2} at the final epoch (mean of 30 seeds):

rrϵr\epsilon_rτr\tau_rmain term
10.420.0262.32
20.200.00225.39
30.150.0008513.3

As rr grows, ϵr\epsilon_r and τr\tau_r decrease and the main term increases. Table 2 in Appendix D.4 shows the same at epochs 500 and 1000.

Is the assumption satisfied? (Appendix D.3–D.4, Fig. 3) The network is pre-fitted to the convex function u∗(x,y,0)=x2+y2/2u^\ast(x,y,0)=x^2+y^2/2, which gives ρθ≤0.075\rho_\theta\le0.075 before training. During training, ρθ<1\rho_\theta<1 held for 54524 of the 60030 networks (90.8%); the violations occurred only within the first 351 epochs, and ρθ=0.369±0.053\rho_\theta=0.369\pm0.053 at the final epoch. Networks with ρθ≥1\rho_\theta\ge1 are excluded from the reported results, since the bound does not apply to them. The result is thus local: it applies to hypothesis classes on which ρθ<1\rho_\theta<1.

Relation to the generalization gap (Fig. 2). Fig. 2 now compares the factor of the bound itself, Q1(θ)Q_1(\theta), with the generalization gap G(θ)=∣Ltest−Ltrain∣G(\theta)=|\mathcal L_{\mathrm{test}}-\mathcal L_{\mathrm{train}}| over 30 seeds, instead of comparing the regularization proxy with the test error. The Spearman correlation is 0.56 (95% confidence interval [0.21, 0.80]). No regularization is used in this experiment, because we do not have a Koopman-based bound for the form (9).

W2/Q2: validation of the regularizer

We redid the Navier–Stokes experiment (Section 5.1, Fig. 1) with 20 runs for each λK∈{0,10−6,10−5,10−4,10−3,10−2}\lambda_K\in\lbrace0,10^{-6},10^{-5},10^{-4},10^{-3},10^{-2}\rbrace instead of the single value 0.010.01, for both the VPINN and the standard PINN.

  • Does the regularizer reduce the factor in the bound? Yes. Fig. 1(a) tracks the factor of Theorem 4.9 itself, ∏l=13∣det⁡Wl∗Wl∣−1/4\prod_{l=1}^{3}|\det W_l^\ast W_l|^{-1/4} (not the proxy), with and without regularization. It decreases monotonically as λK\lambda_K grows, by more than ten orders of magnitude at λK=10−2\lambda_K=10^{-2}.
  • Sensitivity to λK\lambda_K. Fig. 1(b) shows the ratio of the test loss to that for λK=0\lambda_K=0 (geometric mean and 95% confidence interval). For the VPINN, the mean test loss is smaller than that for λK=0\lambda_K=0 for all λK≤10−3\lambda_K\le10^{-3}. For the standard PINN, the result is sensitive to λK\lambda_K and the test loss becomes larger for λK=10−6,10−5,10−2\lambda_K=10^{-6},10^{-5},10^{-2}. Our explanation is that enlarging the determinants can enlarge the upper bound of ∥Kσl∥\Vert K_{\sigma_l}\Vert in Lemma 2.5, so that the product of the two factors in Λθ(i)\Lambda_\theta^{(i)} does not decrease.
  • Size of the improvement. We agree that the gain in test loss is modest and that the confidence intervals are wide. The main purpose of this experiment is to check that the behaviour suggested by the bound is observed.
  • Modifications in Appendix C. For an injective WlW_l, ∣det⁡Wl∗Wl∣1/4=Dldl−1/2|\det W_l^\ast W_l|^{1/4}=D_l^{d_{l-1}/2} exactly, and A~l=1/(1+1/Al)\tilde A_l=1/(1+1/A_l) is a bounded monotone transformation of the upper bound AlA_l of ∥Kσl∥2\Vert K_{\sigma_l}\Vert^2. The regularizer is therefore a surrogate that moves in the same direction as the factor, and Appendix C now says so explicitly.

W3/Q3: determinant term for the output layer

You are right that det⁡W4∗W4=0\det W_4^\ast W_4=0 for W4∈R3×64W_4\in\mathbb R^{3\times64}, so the expression in the theorem is not finite as written. We added a paragraph on this to Appendix C. For a non-injective weight matrix, the factor is to be read in the sense of the bound for non-injective weight matrices of Hashimoto et al. (2026), which is characterized by the orthogonal complement of the kernel of WlW_l; correspondingly, DlD_l is defined through the nonzero singular values only. The regularizer and the factor in Fig. 1(a) involve l=1,2,3l=1,2,3, i.e. W1∈R64×2W_1\in\mathbb R^{64\times2} and W2,W3∈R64×64W_2,W_3\in\mathbb R^{64\times64}, for which the determinant factor was finite in all runs.

W4/Q4: connection to the solution error

We agree, and added this to the limitations (Section 6). Our bound concerns the generalization of the residual, i.e. the gap between the residual loss on the collocation points and its expectation, and not the error of the solution. A small residual implies a small solution error only if the PDE is stable. Under a stability assumption, the solution error is controlled by the residual, and our bound would then control the generalization part of that residual. Making this precise for the variational residual is left for future work.

Reviewer_rhntOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a unified generalization analysis for neural networks trained with differential operators, focusing on PINNs and VPINNs.

For linear differential operators, the authors use adjoints to transfer differentiation from the neural network to smooth test functions, allowing existing Rademacher-complexity bounds to be applied. For nonlinear operators, they introduce Taylor expansions and tensor-product representations to express polynomial nonlinearities as linear operators acting on higher-dimensional spaces.

This construction is subsequently combined with Koopman-based neural-network complexity bounds to investigate the influence of network weights and differential-operator nonlinearities on generalization. Numerical experiments involving steady Navier–Stokes and parabolic Monge–Ampère equations investigate regularization terms motivated by the proposed bounds.

Critical strengths
  1. An interesting approach to extending statistical learning theory to differential residuals.

The paper addresses an important theoretical question: how differential operators involved in PINN and VPINN training affect the statistical complexity of the associated hypothesis classes. The idea of combining adjoint differential operators, tensor lifting, and Koopman-based complexity analysis is conceptually interesting and could motivate further mathematical developments.

  1. A potentially useful distinction between differential-operator complexity and neural-network complexity.

The proposed framework attempts to separate the contributions of the PDE operator from those of the neural-network representation. If established rigorously, this perspective could provide useful insights into how the structure and nonlinearity of differential operators interact with architecture-dependent generalization bounds.

Critical weaknesses

1. The nonlinear lifting construction relies on a diagonal restriction operator that is not closable in the stated Hilbert spaces.

My main concern relates to the mathematical validity of the nonlinear lifting introduced in Section 4.3.

In lines 327–330 (page 7), the authors introduce the diagonal restriction operator PiP^i, formally defined by

(Piv)(x)=v(x,…,x),(P^i v)(x)=v(x,\ldots,x),

which maps functions on a product domain to their restriction to the diagonal.

However, this operator is not well-defined on general L2(Ωi)L^2(\Omega^i) equivalence classes, since the diagonal has measure zero in the product domain. Furthermore, even if PiP^i is defined as a densely defined operator on smooth functions, it is not closable in the L2L^2 topology.

For example, when i=2i=2, one can construct smooth functions vnv_n such that

\qquad P^2v_n\to\varphi\neq0 \quad\text{in }L^2(\Omega).$$ Consequently, the $L^2$ adjoint of this densely defined restriction operator has a trivial domain. In particular, the expressions involving $(P^i)^*$ in Theorems 4.6 and 4.9 cannot be justified as Hilbert-space adjoints acting on the nonzero test functions considered in the paper. A possible remedy would be to work in sufficiently regular Sobolev spaces, where diagonal restriction can be defined as a bounded trace operator. This would allow the nonlinear lifting and its adjoint to be defined rigorously. However, such a modification would require more than simply changing the functional spaces. In particular, the resulting Rademacher-complexity estimates would involve stronger Sobolev norms of the neural-network outputs, whereas the Koopman-based estimates used in Section 4.4 control $L^2$ norms. Therefore, additional analysis would be needed to recover the proposed architecture-dependent bounds, including their dependence on the determinants of the weight matrices. Since this construction is central to the paper's treatment of nonlinear differential operators, I consider this a substantial mathematical concern that affects the validity of the main theoretical contribution. **Additional minor remarks** **a. Applicability to the Monge–Ampère experiment.** In Section 5.2, the authors consider a nonlinear operator involving $\log\det(D^2u)$. However, Theorem 4.9 relies on a Taylor expansion of the nonlinear map at zero, where the logarithmic determinant is not defined. Applying the theorem would require an alternative expansion point, together with appropriate domain and regularity assumptions. It is therefore unclear how the stated theoretical result applies to this experiment. **b. Determinant-based bounds and the experimental architecture.** The Koopman-based bounds in Section 4.4 involve factors of the form $$|\det(W_\ell^\top W_\ell)|^{-1/4}.$$ However, the final weight matrix used in the Navier–Stokes experiment has dimensions $3\times64$, so $W_4^\top W_4$ is necessarily singular. Consequently, the determinant-based expression is not finite for this architecture. Appendix C replaces this quantity with a heuristic based on the positive singular values, but this is not equivalent to the determinant appearing in the theoretical bound. The relationship between the proposed practical regularizer and the theoretical result therefore requires clarification.
Questions for authors
  1. Main concern: mathematical validity Could the authors rigorously justify the diagonal restriction operator (P^i) and the use of its adjoint in Theorems 4.6 and 4.9? In particular, how do they address the non-closability of diagonal restriction in the stated (L^2) spaces? If the intended construction relies on Sobolev trace theory, could the authors specify the appropriate functional spaces and explain how the subsequent Koopman-based generalization bounds would be adapted to the resulting Sobolev norms?

  2. Clarification: Monge–Ampère How does Theorem 4.9 apply to the nonlinear map (g(A)=\log\det(A)), given that its Taylor expansion at zero is not defined? Could the authors specify a valid expansion point and the assumptions required for the corresponding remainder estimate?

  3. Clarification: determinant-based bounds How should the theoretical determinant-based bound be interpreted for the rank-deficient final weight matrix used in the experiments? Can the authors justify that the surrogate regularizer introduced in Appendix C controls the quantity appearing in the theorem, or provide an adapted theoretical bound for this architecture?

2 comments

Thank you for the constructive comments, and in particular for the precise diagnosis of the problem with the diagonal restriction operator. We agree with it. We replaced the construction rather than repairing it. All changes are marked in red in the revised PDF.

Q1 (main concern): validity of PiP^i and its adjoint

You are right that v↦v(x,…,x)v\mapsto v(x,\dots,x) is not well defined on L2(Xi)L^2(\mathcal X^i) and is not closable, so the adjoint (Pi)∗(P^i)^\ast in the original Theorems 4.6 and 4.9 was not justified. The revised paper uses neither functions on the product domain Xi\mathcal X^i, nor a restriction to the diagonal, nor any adjoint of PiP^i.

Pointwise lift (Section 3, Eqs. (5)–(6)). For a measurable w:X→Rd^w:\mathcal X\to\mathbb R^{\hat d} we set Piw⊗i(x)=w(x)⊗i∈(Rd^)⊗iP^iw^{\otimes i}(x)=w(x)^{\otimes i}\in(\mathbb R^{\hat d})^{\otimes i}. This is a function on X\mathcal X itself with values in a finite-dimensional tensor space, so no trace on a null set is involved. It belongs to L2(X,(Rd^)⊗i)L^2(\mathcal X,(\mathbb R^{\hat d})^{\otimes i}) exactly when w∈L2iw\in L^{2i}, with ∥Piw⊗i∥L2=∥w∥L2ii\Vert P^iw^{\otimes i}\Vert_{L^2}=\Vert w\Vert_{L^{2i}}^i, which is finite for bounded ww because X\mathcal X is bounded.

Where adjoints are taken (Assumption 3.2, Eq. (7)). The iith Taylor term is now written as ⟨p,1i!D(i)(0)u⊗i⟩=⟨Ai∗p,Pi(Tu)⊗i⟩\langle p,\tfrac{1}{i!}\mathcal D^{(i)}(0)u^{\otimes i}\rangle=\langle A_i^\ast p,P^i(Tu)^{\otimes i}\rangle, where AiA_i is a differential operator with L∞L^\infty coefficients and Ai∗A_i^\ast is its formal adjoint. The only adjoints therefore act on pxnp_{x_n} (in Section 4.3 it is the adjoint of the finite-dimensional multilinear map g(i)(0)g^{(i)}(0), applied pointwise), Ai∗pxnA_i^\ast p_{x_n} is in L2L^2, and the Cauchy–Schwarz inequality is applied in the Hilbert spaces L2(X,(Rd~)⊗i)L^2(\mathcal X,(\mathbb R^{\tilde d})^{\otimes i}). Example 4.5 was rewritten to verify the assumption explicitly: for the conservative form ∇⋅(u⊗u)\nabla\cdot(u\otimes u) it holds with T=idT=\mathrm{id}, A0=A1=0A_0=A_1=0 and A2∗v=−∇vA_2^\ast v=-\nabla v.

Function spaces. The price of the lift is that the model-side factors in Theorems 4.2 and 4.6 are ∥uθ∥L2ii\Vert u_\theta\Vert_{L^{2i}}^i and ∥D~uθ∥L2ii\Vert\tilde{\mathcal D}u_\theta\Vert_{L^{2i}}^i (i≤ri\le r) instead of L2L^2 norms. No Sobolev norm of the network appears, unlike in the trace-based remedy.

Koopman-based bounds (Section 4.4). You pointed out that the Koopman estimates control only L2L^2 norms. This is resolved by the identity Piuθ⊗i=(Piv⊗i)∘WL∘σL−1∘⋯∘σ1∘W1P^iu_\theta^{\otimes i}=(P^iv^{\otimes i})\circ W_L\circ\sigma_{L-1}\circ\cdots\circ\sigma_1\circ W_1: the lift of the network is again a network of the form (1) with the same weights and activations, and only the final map vv is changed. Lemma 2.6 therefore applies to it directly and gives ∥uθ∥L2ii≤Λθ(i)\Vert u_\theta\Vert_{L^{2i}}^i\le\Lambda_\theta^{(i)} (new Lemma 4.7, proved in Appendix A). Here Λθ(i)\Lambda_\theta^{(i)} in Eq. (11) has the same denominator ∏l∣det⁡Wl∗Wl∣1/4\prod_l|\det W_l^\ast W_l|^{1/4}, and ∥v∥\Vert v\Vert is replaced by ∥v∥L2i(X~L)i\Vert v\Vert^i_{L^{2i}(\tilde{\mathcal X}_L)}. Theorem 4.9 is restated with Λθ(i)\Lambda_\theta^{(i)} under the additional assumption v∈L2iv\in L^{2i}, so the dependence on the determinants is retained for linear D\mathcal D and for the form (8).

Limitation. For the form (9), Theorem 4.6 holds, but D~uθ\tilde{\mathcal D}u_\theta is not of the form (1). The determinants appear only if D~\tilde{\mathcal D} is bounded on L2iL^{2i}, which excludes genuine differential operators. We now state this in Section 4.4 and in the limitations (Section 6), and leave the general case for future work.

Q2: Monge–Ampère

Indeed, g(a,H)=−a+log⁡det⁡Hg(a,H)=-a+\log\det H is not defined at 00. This equation is of the form (9), so the relevant result is Theorem 4.6 rather than Theorem 4.9, and we no longer use the Koopman-based regularizer in this experiment. Section 5.2 was rewritten and Appendix D is new.

  • Expansion point and assumptions (Proposition D.1). Theorem 4.6 is extended to an expansion at any point a0a_0 of an open domain OO of gg, with wθ=D~uθ−a0w_\theta=\tilde{\mathcal D}u_\theta-a_0 in place of D~uθ\tilde{\mathcal D}u_\theta. The assumptions are g∈Cr(O)g\in C^r(O), a0+t wθ(x)∈Oa_0+t\,w_\theta(x)\in O for t∈[0,1]t\in[0,1], wθ∈L2rw_\theta\in L^{2r}, and the pointwise remainder estimate (15). We take a0=(0,A0)a_0=(0,A_0) with A0=D2u∗=diag(2,1)A_0=D^2u^\ast=\mathrm{diag}(2,1) and O={(a,H):H positive definite}O=\lbrace(a,H): H \text{ positive definite}\rbrace.
  • Remainder (Lemmas D.3, D.4, Proposition D.5). Let ρθ\rho_\theta be the supremum over X\mathcal X of the spectral radius of A0−1(D2uθ−A0)A_0^{-1}(D^2u_\theta-A_0). If ρθ<1\rho_\theta<1, the segment condition holds, the Taylor coefficients are explicit, and (15) holds with ϵr=crρθ∥A0−1∥opr/((r+1)(1−ρθ))\epsilon_r=c_r\rho_\theta\Vert A_0^{-1}\Vert_{\mathrm{op}}^r/((r+1)(1-\rho_\theta)), where c1=mc_1=\sqrt m and cr=1c_r=1 for r≥2r\ge2 (here m=2m=2 and ∥A0−1∥op=1\Vert A_0^{-1}\Vert_{\mathrm{op}}=1).

Q2: Monge–Ampère

  • Is the assumption satisfied? (Appendix D.3–D.4, Fig. 3) The network is pre-fitted to the convex function u∗(x,y,0)=x2+y2/2u^\ast(x,y,0)=x^2+y^2/2, which gives ρθ≤0.075\rho_\theta\le0.075 before training. During training, ρθ<1\rho_\theta<1 held for 54524 of the 60030 networks (90.8%); the violations occurred only within the first 351 epochs, and ρθ=0.369±0.053\rho_\theta=0.369\pm0.053 at the final epoch. D2uθD^2u_\theta was positive definite at all quadrature nodes for all networks. Networks with ρθ≥1\rho_\theta\ge1 are excluded from the reported results, since the bound does not apply to them.

Q3: determinant-based bounds and the rank-deficient output layer

We added a paragraph on this gap to Appendix C and changed the experiment in Section 5.1 so that it reports the factor of the theorem.

  • Interpretation for W4∈R3×64W_4\in\mathbb R^{3\times64}. You are right that det⁡W4∗W4=0\det W_4^\ast W_4=0, so the expression in Theorem 4.9 is not finite as written. For a non-injective weight matrix, the factor is to be read in the sense of the bound for non-injective weight matrices of Hashimoto et al. (2026), which is characterized by the orthogonal complement of the kernel of WlW_l. Correspondingly, DlD_l in Appendix C is defined through the nonzero singular values only.
  • Relation between the surrogate and the theorem. If WlW_l is injective, then ∣det⁡Wl∗Wl∣1/4=Dldl−1/2|\det W_l^\ast W_l|^{1/4}=D_l^{d_{l-1}/2} exactly, so Dl−1/2D_l^{-1/2} is a monotone function of the determinant factor; they differ by the normalization with the rank. Likewise, A~l=1/(1+1/Al)\tilde A_l=1/(1+1/A_l) is a bounded monotone transformation of the upper bound AlA_l of ∥Kσl∥2\Vert K_{\sigma_l}\Vert^2 from Lemma 2.5. We now state explicitly that the regularizer does not coincide with the factor in the bound and is a surrogate for it.
  • Evidence that the surrogate controls the factor. The regularizer is λK∑l=13A~l/Dl1/2\lambda_K\sum_{l=1}^{3}\tilde A_l/D_l^{1/2}, over W1∈R64×2W_1\in\mathbb R^{64\times2} and W2,W3∈R64×64W_2,W_3\in\mathbb R^{64\times64}. The new Fig. 1(a) reports the factor of the theorem itself, ∏l=13∣det⁡Wl∗Wl∣−1/4\prod_{l=1}^{3}|\det W_l^\ast W_l|^{-1/4}, at the final epoch for λK∈{0,10−6,…,10−2}\lambda_K\in\lbrace0,10^{-6},\dots,10^{-2}\rbrace (20 runs each). It is finite in all cases and decreases monotonically as λK\lambda_K grows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.