- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper views full-batch PINN training with hard-constrained PINN loss, whose curvature is the Gauss–Newton (GN) matrix . It will be hard to solve when is ill-conditioned. The authors experimentally found that the input and output layer controls most of the ill-condition of , but only holds a very small part of the whole network parameters. So they planed to only compute the exact Gauss–Newton blocks of input/output layers and uses them to precondition a fixed-budget conjugate-gradient (CG) solve while the other layers stay matrix-free. This significantly reduced the ill-condition of and significantly reduced the computational time of computing the inverse of . Numerous experiments are conducted to support the method.
Critical strengths
1.LANCE is a simple, cheap, and analytically clean method. It is a drop-in preconditioner inside an existing matrix-free GN-CG loop. The small block preconditioner is fast to compute and reduce the ill-condition significantly.
2.The measurement study is well executed, and the within-scope evidence is strong. The finding that input/output layer dominates the curvature energy is interesting.
Critical weaknesses
1.While the finding that input/output layer dominates the curvature energy is interesting, it is experimentally founded rather than theoretically estimated. Although the authors provide some explanations in Appendix C.4, they are not strictly evidence. Notice that the global convergence of natural gradient method for training PINNs has been established in the last year ICLR [1], it is suggested to give more theoretical estimate or explanation.
[1]Fast Convergence of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks. ICLR 2026.
2.The evidence is confined to a narrow regime, every main experiment is full-batch, hard-constrained, float64, JAX, one tanh MLP family on manufactured solutions. For example, FP64 may differs significantly with FP32 (FP64 is all you needed, NeurIPS 2025), experiments should meet a larger criterion. The paper's own scaling data also show the method's premise degrading exactly where PINNs are most attractive. For example, for higher dimensional problems (5D, 10D, and not tested 100D cases), the gains drops significantly.
3.The writing should be more specific and easier to understand. Some AI-polished text in the paper is hard to read. For example, I couldn't understand what the authors were trying to say in the conclusions and limitations section.
Questions for authors
1.Does LANCE holds for natural gradient method, where the gram matrix is computed by in above reference [1], rather than in your paper?
2.Could you report error-vs-wall-clock (or error-vs-products) curves in which the baseline is allowed the same total budget as LANCE, in particular GN-CG at (currently ahead at equal iterations but using the products per step) and CG-to-tolerance?
3.What about the influence of FP64 and FP32 in LANCE?
4.What about the gains of LANCE for very high-dimensional probelms? Inverse Problems? These are regions where PINNs overcomes traditional numerical methods, and only these problems are more valuable to be studied.
5.Proposition 3 assumes ; every experiment has . What happens for (small networks or large collocation sets), and with soft constraints plus adaptive loss weights, non-tanh activations/Fourier features? These are common in PINN training literatures.
Thank you for asking whether the concentration in the end layers can be estimated theoretically, and whether it persists outside our original training setup. We address the density estimate first and then the cases you raised.
Trace density and natural gradient. Proposition 5 gives a conditional estimate. If the traces of the L layer blocks are within a factor c of one another, the density of layer ℓ is within a factor c of P/(LPℓ), where Pℓ is its parameter count. Thus, layers with comparable traces can have quite different curvature per parameter. For example, on Helmholtz-1D at width 64 the estimate gives densities of 16.6 and 32.7 for the input and output layers, compared with measured values of 14.8 and 23.0 at initialisation (Table 25). One clarification is that LANCE improved the short CG solve even though the global condition number increased (Section 4.1).
For our discretised least-squares loss, the quadrature weights are included in the residual Jacobian J. The parameter-space Gram matrix is then J⊤J, so LANCE can be used for this damped natural-gradient system. If the system is instead written in residual space using JJ⊤, the corresponding parameter update can be recovered by applying J⊤ to the residual-space solution. Our layer blocks act on the parameter-space system. We now cite Xu et al. in Section 1 and distinguish their convergence result for an exact natural-gradient step from our measurements with a finite CG budget.
Wall-clock time and precision. Figure 13 and Table 28 compare LANCE with GN-CG at 10, 20 and 40 inner iterations and with CG run to tolerance. At the same total training time, LANCE achieved lower errors than the best fixed-budget GN-CG run on the two Burgers problems and Poisson-2D. On Helmholtz-1D, GN-CG with 20 inner iterations gave a 4.1-fold lower error.
In float32, LANCE reduced the error relative to GN-CG by factors of 1.6 and 1.9 on the two Burgers problems, gave essentially the same error on Poisson-2D, and reduced the error by a factor of 35 on Helmholtz-1D (Table 40, five seeds). On Helmholtz-1D the larger ratio arose mainly because GN-CG deteriorated in float32. LANCE's error was about 5×10−3 in both precisions. We also raised the damping floor and block jitter to 10−6 for the float32 runs, so this comparison includes those changes.
Dimension and number of collocation points. On Poisson-10D, LANCE and GN-CG both finished at a relative error of 0.113. Raising the size cap to admit the input block did not change that result (Table 22). We have no results for 100-dimensional or inverse problems. We did test systems with more residuals than parameters on Burgers and Helmholtz-1D (Table 30). LANCE gave lower errors on both problems with ten inner iterations. With 40, GN-CG gave lower errors at width 32 and the two methods were close at width 64. We did not measure the near-zero spectrum in these runs, so the spectral explanation for the original experiments remains unverified here.
Constraints, activations and exposition. With fixed soft boundary penalties, the initial layer trace shares changed by at most 0.007 on the two problems tested. Training outcomes differed considerably: LANCE improved on GN-CG on Poisson-2D, while both methods retained large errors on Helmholtz-1D (Table 24). We did not test adaptive loss weights. Appendix C.5 reports the initial density measurements for GELU and softplus, and Table 32 gives training results with Fourier features. In revising Sections 6 and 7, we have distinguished the quantity estimated by the theory from the practical result: the end layers offer an inexpensive improvement over unpreconditioned CG, but the rank-50 sketch gave lower errors on all four primary problems.