acceptodds
Under review as a conference paper at ICLR 2027

LANCE: Layer-Aware Preconditioning for Gauss-Newton Training of PINNs

Abstract

Physics-informed neural networks (PINNs) minimise the squared norm of a collocation residual, a deterministic nonlinear least-squares problem whose curvature is inherited from a differential operator and whose condition number reaches . First-order and limited-memory quasi-Newton methods stall on it, and a line of work has moved PINN training towards Newton and natural-gradient methods that differ mainly in how much of the curvature matrix they represent. Within that line, we ask which part of the curvature is worth representing exactly when only a few iterations of the inner linear solve can be afforded per step by measuring the curvature layer by layer. Across nine PDEs and networks of up to parameters, the input and output layers hold to of the parameters and to of the curvature energy. No scalar per layer preconditioner can exploit this concentration, because scaling a block by a constant leaves its internal conditioning unchanged. LANCE forms the exact Gauss–Newton blocks of those layers and uses them to precondition a fixed-budget conjugate-gradient (CG) solve while the other layers stay matrix-free. Once the damping has fallen, ten CG iterations capture to of the attainable model decrease without the preconditioner and to with it. As a result, the final relative error is to times lower than without the preconditioner, at up to times the time per step.

Primary area:
optimization
Keywords:
physics-informed neural networks, PINNs, scientific machine learning, second-order optimization, Gauss-Newton, natural gradient descent, preconditioning, conjugate gradient, curvature approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QNxyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper views full-batch PINN training with hard-constrained PINN loss, whose curvature is the Gauss–Newton (GN) matrix G=JTJG=J^T J. It will be hard to solve when GG is ill-conditioned. The authors experimentally found that the input and output layer controls most of the ill-condition of GG, but only holds a very small part of the whole network parameters. So they planed to only compute the exact Gauss–Newton blocks of input/output layers and uses them to precondition a fixed-budget conjugate-gradient (CG) solve while the other layers stay matrix-free. This significantly reduced the ill-condition of GG and significantly reduced the computational time of computing the inverse of GG. Numerous experiments are conducted to support the method.

Critical strengths

1.LANCE is a simple, cheap, and analytically clean method. It is a drop-in preconditioner inside an existing matrix-free GN-CG loop. The small block preconditioner is fast to compute and reduce the ill-condition significantly.

2.The measurement study is well executed, and the within-scope evidence is strong. The finding that input/output layer dominates the curvature energy is interesting.

Critical weaknesses

1.While the finding that input/output layer dominates the curvature energy is interesting, it is experimentally founded rather than theoretically estimated. Although the authors provide some explanations in Appendix C.4, they are not strictly evidence. Notice that the global convergence of natural gradient method for training PINNs has been established in the last year ICLR [1], it is suggested to give more theoretical estimate or explanation.

[1]Fast Convergence of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks. ICLR 2026.

2.The evidence is confined to a narrow regime, every main experiment is full-batch, hard-constrained, float64, JAX, one tanh MLP family on manufactured solutions. For example, FP64 may differs significantly with FP32 (FP64 is all you needed, NeurIPS 2025), experiments should meet a larger criterion. The paper's own scaling data also show the method's premise degrading exactly where PINNs are most attractive. For example, for higher dimensional problems (5D, 10D, and not tested 100D cases), the gains drops significantly.

3.The writing should be more specific and easier to understand. Some AI-polished text in the paper is hard to read. For example, I couldn't understand what the authors were trying to say in the conclusions and limitations section.

Questions for authors

1.Does LANCE holds for natural gradient method, where the gram matrix is computed by G=JJTG=J J^T in above reference [1], rather than G=JTJG=J^T J in your paper?

2.Could you report error-vs-wall-clock (or error-vs-products) curves in which the baseline is allowed the same total budget as LANCE, in particular GN-CG at K=40K=40 (currently ahead at equal iterations but using ∼3.5×\sim3.5\times the products per step) and CG-to-tolerance?

3.What about the influence of FP64 and FP32 in LANCE?

4.What about the gains of LANCE for very high-dimensional probelms? Inverse Problems? These are regions where PINNs overcomes traditional numerical methods, and only these problems are more valuable to be studied.

5.Proposition 3 assumes N<PN<P; every experiment has N/P≲0.5N/P \lesssim 0.5. What happens for N≥PN\ge P (small networks or large collocation sets), and with soft constraints plus adaptive loss weights, non-tanh activations/Fourier features? These are common in PINN training literatures.

1 comment

Thank you for asking whether the concentration in the end layers can be estimated theoretically, and whether it persists outside our original training setup. We address the density estimate first and then the cases you raised.

Trace density and natural gradient. Proposition 5 gives a conditional estimate. If the traces of the LL layer blocks are within a factor cc of one another, the density of layer ℓ\ell is within a factor cc of P/(LPℓ)P/(L P_\ell), where PℓP_\ell is its parameter count. Thus, layers with comparable traces can have quite different curvature per parameter. For example, on Helmholtz-1D at width 64 the estimate gives densities of 16.6 and 32.7 for the input and output layers, compared with measured values of 14.8 and 23.0 at initialisation (Table 25). One clarification is that LANCE improved the short CG solve even though the global condition number increased (Section 4.1).

For our discretised least-squares loss, the quadrature weights are included in the residual Jacobian JJ. The parameter-space Gram matrix is then J⊤JJ^\top J, so LANCE can be used for this damped natural-gradient system. If the system is instead written in residual space using JJ⊤JJ^\top, the corresponding parameter update can be recovered by applying J⊤J^\top to the residual-space solution. Our layer blocks act on the parameter-space system. We now cite Xu et al. in Section 1 and distinguish their convergence result for an exact natural-gradient step from our measurements with a finite CG budget.

Wall-clock time and precision. Figure 13 and Table 28 compare LANCE with GN-CG at 10, 20 and 40 inner iterations and with CG run to tolerance. At the same total training time, LANCE achieved lower errors than the best fixed-budget GN-CG run on the two Burgers problems and Poisson-2D. On Helmholtz-1D, GN-CG with 20 inner iterations gave a 4.1-fold lower error.

In float32, LANCE reduced the error relative to GN-CG by factors of 1.6 and 1.9 on the two Burgers problems, gave essentially the same error on Poisson-2D, and reduced the error by a factor of 35 on Helmholtz-1D (Table 40, five seeds). On Helmholtz-1D the larger ratio arose mainly because GN-CG deteriorated in float32. LANCE's error was about 5×10−35\times10^{-3} in both precisions. We also raised the damping floor and block jitter to 10−610^{-6} for the float32 runs, so this comparison includes those changes.

Dimension and number of collocation points. On Poisson-10D, LANCE and GN-CG both finished at a relative error of 0.113. Raising the size cap to admit the input block did not change that result (Table 22). We have no results for 100-dimensional or inverse problems. We did test systems with more residuals than parameters on Burgers and Helmholtz-1D (Table 30). LANCE gave lower errors on both problems with ten inner iterations. With 40, GN-CG gave lower errors at width 32 and the two methods were close at width 64. We did not measure the near-zero spectrum in these runs, so the spectral explanation for the original experiments remains unverified here.

Constraints, activations and exposition. With fixed soft boundary penalties, the initial layer trace shares changed by at most 0.007 on the two problems tested. Training outcomes differed considerably: LANCE improved on GN-CG on Poisson-2D, while both methods retained large errors on Helmholtz-1D (Table 24). We did not test adaptive loss weights. Appendix C.5 reports the initial density measurements for GELU and softplus, and Table 32 gives training results with Fourier features. In revising Sections 6 and 7, we have distinguished the quantity estimated by the theory from the practical result: the end layers offer an inexpensive improvement over unpreconditioned CG, but the rank-50 sketch gave lower errors on all four primary problems.

Reviewer_mwBiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LANCE, a layer-aware preconditioning method for Gauss–Newton training of physics-informed neural networks (PINNs). Based on each layer’s curvature trace density and parameter count, LANCE selects a subset of small layers for exact curvature-block construction and uses scalar approximations for the remaining layers. It then applies preconditioned conjugate gradient (PCG) with a fixed iteration budget to compute updates while retaining full inter-layer coupling. Across multiple PDE benchmarks and ablation studies, the paper reports improved final accuracy under the same limited CG iteration budget.

Critical strengths

The paper presents a practical approach to improving the quality of approximate Gauss–Newton updates under a limited CG budget.

The experimental evaluation covers multiple PDE problems and includes extensive ablation studies. The authors also discuss negative results and limitations rather than reporting only favorable outcomes.

Critical weaknesses

The authors observe that the input and output layers have higher curvature trace density, but the connection between trace density and the difficulty of solving the Gauss–Newton system is not clearly established. In particular, high trace density does not directly imply that a layer is difficult for CG to resolve or that representing its curvature block exactly is the most effective use of computational resources. Furthermore, the introduction emphasizes poor conditioning as a central challenge in PINN training, whereas the experiments show improved accuracy despite a worse global condition number under LANCE.

Given the issues with the motivation, I consider this work a heuristic method. For a heuristic method, it is necessary to demonstrate that other simple methods cannot achieve performance comparable to LANCE. Specifically, the experiments should include random layer selection, fixed selection of the input and output layers, and fixed strategies such as selecting intermediate layers.

The experimental section states that comparisons use the same number of CG iterations, but LANCE increases computation time. Moreover, when the total number of steps is large, LANCE does not achieve the best performance. Therefore, experiments under the same computation time budget are needed.

Several symbols and abbreviations are introduced without sufficient explanation; for example, ENGD in Figure 1 is not clearly introduced beforehand. In addition, lines 75–82 of the introduction rely heavily on experimental observations before the experimental setup has been explained. This makes the central narrative difficult to follow without repeatedly consulting later sections.

Questions for authors

The main empirical analyses and training experiments are based on standard MLPs. Do the observed trace-density patterns and the benefits of the proposed selection rule persist with residual connections, separable architectures such as SPINN, or architectures that regulate layerwise Jacobian properties, such as CoupledNet? This would help clarify whether the empirical motivation and practical benefits of LANCE extend beyond the tested MLP family.

1 comment

Thank you for suggesting the fixed and random layer comparisons. They allowed us to test whether the input and output layers were particularly useful, or simply inexpensive to include. The distinction changed our interpretation of the method.

Which layers should be selected? We compared the density rule with three alternatives: always selecting the input and output layers, always selecting the two hidden layers, and drawing a random pair once per seed. We also tested the rule without the size cap and a preconditioner using all four blocks (Table 27, ten seeds). At the same number of training iterations, the fixed hidden pair and the random pair both gave lower errors than the default rule on all three problems. Using the hidden pair cost 5.7–7.8 times as much as unpreconditioned GN-CG, compared with 1.0–1.2 times for the default rule. The advantage of selecting the end layers is that their blocks are inexpensive. We found no evidence that they are the optimal pair at a fixed iteration count.

Trace density and conditioning. We agree that high trace density does not establish that CG has difficulty resolving a layer. We therefore measured how much additional model decrease was available when we corrected one layer at a time after CG stopped. The two end layers contained 2–3% of the parameters but accounted for 29–49% of the sum of the layerwise improvements after initialisation (Proposition 4 and Table 29). This measurement gives direct evidence for using the small blocks, while the trace remains a heuristic for selecting them.

Your observation about the global condition number is also correct: it was larger under LANCE. Unpreconditioned CG starts from zero and stays in the range of J⊤J^\top, so eigenvalues on the null space of JJ do not affect its iterates. Proposition 6 gives the condition under which a preconditioner preserves this range. For LANCE, 73–99% of the squared length of the direction lay in the numerical null space in our measurements (Figure 3). We therefore assess the accuracy of the inner solve by the model decrease it achieves after a fixed number of iterations (Sections 4.1 and 5.5).

Equal training time. Figure 13 plots relative L2L^2 error against elapsed training time. Table 28 includes unpreconditioned CG with 10, 20 and 40 inner iterations, as well as CG run to a residual tolerance. At the LANCE time budget, LANCE's final errors were lower than those of the best fixed-budget GN-CG run by factors of 2.7 on each Burgers problem and 2.1 on Poisson-2D. On Helmholtz-1D, GN-CG with 20 inner iterations gave a 4.1-fold lower error.

Other architectures and presentation. On residual MLPs, LANCE reduced the final error relative to GN-CG by factors of about 13 on Helmholtz-1D and 22 on Burgers. The size cap mattered more for the Fourier-feature MLP and reduced PirateNet: their 4,160-parameter input blocks exceeded the default cap of 512. With this cap, the Fourier-feature model benefited on Helmholtz-1D, but the other three architecture–problem combinations showed little improvement or a higher error (Table 32). We did not test SPINN or CoupledNet, so the applicability of the rule to those architectures remains open. We have also expanded the method names in Section 1. Figure 1 compares the curvature representations, and Section 3 gives the measurements used to motivate layer selection.

Reviewer_6ETBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LANCE, an optimizer for PINNs. The authors first observe that the input and output layers typically have disproportionately higher curvature, as quantified by the proportion of the trace energy of the Gauss-Newton matrix. This motivates LANCE, which computes the Gauss-Newton blocks for those high-curvature parameters and uses it as a preconditioner for the conjugate gradient (CG) method. Optimizing with CG and LANCE's preconditioner achieves 4.5-25x lower L2RE than without that preconditioner, while increasing wall-clock time by only 6-38%.

Critical strengths
  • Improving the efficiency of second-order optimization methods for PINNs is an important problem, given the severe ill-conditioning that often arises in PINN training. The paper proposes a relatively simple and intuitive approach to preconditioning Gauss–Newton CG, exploiting the fact that certain layers are small enough for their curvature blocks to be computed exactly. The empirical results demonstrate substantial improvements over unpreconditioned GN-CG on several PDE problems, with relatively modest additional computational overhead.
Critical weaknesses
  • The proposed layer-selection criterion appears largely heuristic, without a clear theoretical justification for why it identifies the parameters most valuable to precondition. In particular, the authors use the trace of the Gauss–Newton blocks to identify the input and output layers as disproportionately important. However, high curvature density does not necessarily imply poor conditioning within those blocks, nor does it establish that the error from truncated CG is concentrated in those parameter blocks. While the empirical results suggest that preconditioning these layers can be effective, the theoretical results do not establish a direct connection between the proposed selection criterion and the reduction in CG truncation error. This leaves a gap in the central theoretical motivation for the method.
  • The experiments are restricted primarily to standard tanh MLP architectures and relatively simple PDE problems with manufactured solutions. Moreover, the main comparisons against independently implemented optimizers are limited to Adam and L-BFGS, while comparisons with more advanced second-order methods use simplified implementations rather than the published algorithms. The proposed method also provides limited or no improvement on several of the tested PDEs. Given the availability of stronger PINN optimizers, more recent architectures, and more challenging benchmarks, it is unclear how broadly the reported improvements generalize. See the questions above for specific suggested comparisons.
Questions for authors
  • The authors compare to CG without preconditioning and a layer-scalar preconditioner. How well does LANCE perform compared to a slightly more complex, Jacobi preconditioner, formed by M_diag = diag(G) + \lambda I? This can change the condition number within blocks, so is not ruled out by Proposition 1, and does not require additional computation given the algorithm already computes the entire diagonal of G during its per-point gradient pass.
  • Similarly, could the authors compare to stronger PINN optimizers, such as NN-CG [1] and SSBroyden [2]?
  • Do the improvements in relative error persist with newer PINN architectures, such as PirateNet [3], and more challenging PDE tasks, such as from [4, 5]?

References: [1] Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML, 2024. [2] Kiyani et al. Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks. Computer Methods in Applied Mechanics and Engineering, 2025. [3] Wang et al. PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. Journal of Machine Learning Research, 2024. [4] Hao et al. PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs. NeurIPS, 2024. [5] Wang et al. An Expert's Guide to Training Physics-informed Neural Networks. arXiv:2308.08468, 2023.

1 comment

Thank you for suggesting point-Jacobi, NNCG and self-scaled Broyden. LANCE improved on the diagonal preconditioner, but the Nyström preconditioner and self-scaled Broyden both gave lower errors than LANCE in the comparisons you requested.

Layer selection and point-Jacobi. We agree that the trace of a block does not establish how well conditioned it is. We now treat trace density as a selection heuristic and measure the effect of a layer directly: after CG stops, we compute the largest additional decrease in the quadratic model that a correction confined to that layer can achieve (Proposition 4 and Table 29). The input and output layers offer a large improvement per parameter, although the hidden layers together account for more of the sum of the layerwise improvements.

Point-Jacobi is a particularly useful control because it rescales parameters within each layer and the diagonal is already available from the per-point gradient pass. It improved on unpreconditioned CG on five of the nine problems. LANCE still gave a lower final error than point-Jacobi on all seven benchmarks meeting the accuracy criterion, by factors of 1.8–43 at the same number of training iterations (Table 26).

Nyström preconditioning and self-scaled Broyden. To isolate the choice of preconditioner, we used NNCG's Nyström construction in the same damped Gauss–Newton training loop as LANCE. We did not run the complete Adam/L-BFGS/Newton-CG training schedule of Rathore et al. With rank 50 and a refresh every ten training iterations, the final errors were 2.2–52 times lower than with LANCE on the four primary problems, at a similar cost per iteration (Table 31). Increasing the rank to 200 gave no consistent improvement. Combining the sketch and the blocks also gave higher errors than the sketch alone at the same number of matrix–vector products. A separate diagnostic showed that ten CG iterations with the rank-50 preconditioner captured 99.7–100% of the attainable model decrease (Table 36).

We also implemented self-scaled Broyden following Kiyani et al., using a dense inverse-Hessian approximation and a strong Wolfe line search. At the LANCE wall-clock budget, it achieved a lower final error on all seven benchmarks meeting the accuracy criterion (Table 33). We then gave it the time taken by 300 training iterations of the rank-50 solver with an Armijo line search. At this budget, its errors were 11–38 times lower on the four primary problems (Table 39). These results favour self-scaled Broyden at the network size used in the main benchmark. Its dense inverse-Hessian approximation requires storage quadratic in the number of parameters.

Architectures and harder problems. The reduced PirateNet did not benefit from LANCE with the default size cap on either Helmholtz-1D or Burgers. Its input block has 4,160 parameters. When we allowed blocks of that size, the final errors decreased by factors of about 51 and 35 relative to unpreconditioned CG, respectively (Table 32). This is a materially more expensive preconditioner. We also report results for residual and Fourier-feature MLPs in the same table. The remaining limitations are important: Helmholtz-2D and Poisson-10D remained difficult, and on Wave-1D both Adam and L-BFGS achieved lower errors than LANCE (Table 1). We have not evaluated PINNacle or the broader benchmark suite suggested in the review. The comparisons with KFAC-PINN, D-NGD and ENGD use our implementations of their curvature representations rather than the published training codes.

Comments on the paper

1 comment

Thank you for asking us to test the reasoning behind our choice of layers. We originally argued that the input and output layers were especially useful to precondition because they contain few parameters but account for a large share of the Gauss–Newton trace. As the reviewers pointed out, the trace alone does not tell us which parts of a CG step are still inaccurate. We have revised that argument and added the comparisons needed to assess it.

We first measured how much further the quadratic model could be reduced after ten CG iterations if we allowed a correction to just one layer. Repeating this calculation for each layer gave us a common basis for comparison (Proposition 4). The input and output layers together contained 2–3% of the parameters and accounted for 29–49% of the sum of these individual improvements after initialisation (Table 29). This explains why two small blocks can be useful. It also qualifies our original interpretation: the hidden layers together offered a larger improvement, although their blocks cost considerably more to use (Table 27).

The reviewers also asked whether it would be better to spend the extra computation on more CG iterations. We gave unpreconditioned CG the same total training time as LANCE and selected its best iteration budget for each problem. LANCE reduced the final relative L2L^2 error by factors of 2.1–2.7 on the two Burgers problems and Poisson-2D. On Helmholtz-1D, unpreconditioned CG gave a 4.1-fold lower error (Figure 13 and Table 28).

The strongest comparison was with a rank-50 Nyström preconditioner. It gave a lower final error than LANCE on all four primary problems at a similar cost per training iteration (Table 31). The spectral measurements help explain this result. Across these problems, the leading 50 eigendirections accounted for 98–100% of the decrease attainable in the quadratic model (Table 36). The layer blocks also introduced a substantial component outside the range of the transposed Jacobian: 73–99% of the squared length of the LANCE direction lay in the numerical null space in these measurements (Figure 3 and Table 34). The sketch gave a lower final error in five of the six larger configurations tested (Table 38). At approximately 8,500 parameters, where a dense inverse-Hessian approximation fitted in memory, self-scaled Broyden outperformed LANCE on all seven benchmarks meeting our accuracy criterion at matched training time (Table 33).

Our conclusion is therefore narrower than in the submission. Exact blocks for the input and output layers improve a short CG solve at modest cost in the networks studied here. When choosing a preconditioner for these benchmarks, however, the results favour the low-rank approximation of the full matrix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.