LANCE: Layer-Aware Preconditioning for Gauss-Newton Training of PINNs

Physics-informed neural networks (PINNs) minimise the squared norm of a collocation residual, a deterministic nonlinear least-squares problem whose curvature is inherited from a differential operator and whose condition number reaches $10^{15}$. First-order and limited-memory quasi-Newton methods stall on it, and a line of work has moved PINN training towards Newton and natural-gradient methods that differ mainly in how much of the curvature matrix they represent. Within that line, we ask which part of the curvature is worth representing exactly when only a few iterations of the inner linear solve can be afforded per step by measuring the curvature layer by layer. Across nine PDEs and networks of up to $2.6\times 10^5$ parameters, the input and output layers hold $0.3$ to $15\%$ of the parameters and $14$ to $42\%$ of the curvature energy. No scalar per layer preconditioner can exploit this concentration, because scaling a block by a constant leaves its internal conditioning unchanged. LANCE forms the exact Gauss–Newton blocks of those layers and uses them to precondition a fixed-budget conjugate-gradient (CG) solve while the other layers stay matrix-free. Once the damping has fallen, ten CG iterations capture $0.03$ to $0.06$ of the attainable model decrease without the preconditioner and $0.38$ to $0.97$ with it. As a result, the final relative ${\mathrm{L}}^2$ error is $4.5$ to $25$ times lower than without the preconditioner, at up to $1.4$ times the time per step.