- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript proposes a iteratively reweighed least squares-like modification of the PDE term in a PINN loss. The idea is interesting, but the manuscript has two significant weaknesses: (i) It does not use Gauss-Newton type (or natural gradient in ML parlance) methods for optimization, although it is by now well-established that these are orders of magnitude more accurate than Adam or L-BFGS, (ii) it tests the method on much too simple problems. These problems are fundamental and lead me to recommend rejection. I give more details below, and a suggestion on how to use the paper's idea in a Gauss-Newton context.
Technical summary The plain PINN loss is
The manuscript proposes to re-weight the PDE term to be
$ ||r_{\text{PDE}}(\theta) ||_{W(\text{sg}(\theta))}^2 $
where is a block-diagonal matrix (with small blocks) depending on through the jet of and is the stop gradient operator. The re-weighting matrix is obtained in the following way:
For a fixed collocation point the point-wise satisfaction of the PDE is encoded via the function taking spatial coordinate and the jet as input and returning in
The plain PINN loss uses (the sum over) . There are however several choices of F that lead to the same PDE. This redundancy leads the authors to propose a geometric variant in place of the squared L2 loss based on the distance to the zero set of the PDE
where . To make this tractable the constraint is linearized at the current jet which turns the problem into a quadratic minimization over an affine linear subspace (the linearization of the manifold) and this has a closed form solution. Denoting by the Jacobian of with respect to at , this approximation can be computed as (assuming full rank of )
$ F_{x_i}(\xi_i)^T (J_i J_i^\top)^{-1} F_{x_i}(\xi_i) $
So setting as a block in the block diagonal weighting matrix is the formula above.
Critical strengths
Addressing freedom in the choice of for the PDE is an interesting idea.
Critical weaknesses
There are two critical weaknesses:
1.) The manuscript does not use state of the art optimizers for PINNs. Gauss-Newton/natural gradient type methods perform orders of magnitude better for essentially all known instances in PINN training, while at the same time remaining scalable and efficient. GN optimization often takes a fraction of the time of Adam [ENGD, DeepMind, Karniadakis]. Benchmarking a new loss function with an outdated training protocol is therefore not meaningful, and unfortunately weakens the empirical findings significantly.
2.) The paper uses very easy benchmark problems that can be solved to high accuracy with Gauss-Newton and the plain PINN formulation. I prompted Claude Opus 5.5 with reasoning on high through the web API in the following way:
"Can you take the examples of this paper, use JAX in double precision and a kernel formulation of Levenberg Marquardt with constant damping of 1e-5 and backtracking line search with Armijo and quickly run their experiments and give me the relative L2 error?"
The results were: Wave relative L2 3.68e-5, Reaction-Diffusion relative L2 1.89e-4, Kovasznay relative L2 3.3e-4.
While I do not intent to check Claude's code carefully, this is exactly what I expected and more accurate than all runs in this paper.
Nevertheless, I think there might be something interesting to the core idea of the paper and I give the details here: Applying Levenberg-Marquardt (=Gauss-Newton with damping) in kernel formulation to the weighted PDE residual (dropping other terms for simplicity) gives the update formula (after some sorting of terms)
where in this case is the Jacobian of the residual with respect to the trainable parameters (sorry for the clash with your notation). The is your weighting matrix. So what this yields is a principled way to substitute the unmotivated identity based damping that is very often used. I leave it to the authors to pursue this idea---it is certainly not ment as a suggestion for what has to be done in a revision, as it will lead to a significantly different paper.
Literature:
[ENGD] https://proceedings.mlr.press/v202/muller23b/muller23b.pdf
[DeepMind] https://arxiv.org/html/2509.14185v1
[Karniadakis] https://arxiv.org/pdf/2604.05230
Questions for authors
I have no questions.