- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper builds upon multihead architectures for PINNs in parametric PDEs and different initial conditions. First, the basis and heads are trained together on a set of instances of the parameters (or ICs). Then, the basis is frozen. Unlike other works, this paper adopts two second-order optimization procedures for the head on new instances, which is especially useful for nonlinear PDEs. With Hessian continuation, the authors are also able to obtain paths in the head space based on the parameters. The results are very good compared to first-order optimization based on Adam.
Critical strengths
- The paper is well written and easy to follow, with a clear presentation of the problem. The main text reports almost all relevant information and steps to understand the methodology and the results. Overall, the paper is in very good shape.
- The ideas and methodology are sound, and the different choices are justified.
- The results are good, and the visualizations are also very helpful. The paper also reports some useful ablations.
- The idea of using a VAE for the different IC is very nice
Critical weaknesses
- I have some concerns only about some choices and possible ablations required, mainly regarding baselines and other details.
Questions for authors
- What is the motivation behind the choice of for the VAE? I understand this dimension shouldn't be too high, but the specific choice might depend heavily on the problem.
- The initial condition is calculated with a sum of sine functions. This only works for problems with null Boundary conditions. What can be done in other cases? Are there experiments for this?
- I am unsure about the choice of using an orthonormal basis. I understand some of the motivation, but not everything, and not the concrete advantages. Could the authors provide more comments on why it is useful and what happens if you do not use it? Also, an ablation without this constraint could be nice.
- Related to the previous point: on lines 255-256, the phrase "With varying ICs, Aj changes too and head distances measure the part of the solution carried by the head only." is a bit unclear. Please provide more details.
- The choice of Adam as a baseline is fair, but I am also interested in seeing what happens with L-BFGS, as it has clear advantages on a lot of PINN-related problems.
- On line 312: is not clear to me.
- I would report some insights from the results on Adam without the same wall-clock time (Appendix B6 and Table 3) in the main text, at least to understand the full comparison.
- In Figure 3, the known heads are already very dense, so interpolation might be easy. What happens if there is a lot of space in between or if we extrapolate far away?
On a final note, Images are very small.
In general, the paper is sound and already in good shape, and I lean towards acceptance with possible increase in the score
4 comments
Orthonormal basis and its ablation
Why an orthonormal basis, and what happens without it?
The motivation is twofold: it fixes the head of each instance uniquely, and it makes the weights readable as coordinates (see the next point). We ran an ablation on the four multi-IC models: no penalty (), the value of the paper () and a stronger one (), with three training seeds for the first two. It is reported in the new Appendix H.1 (Fig. 12). Between and neither the accuracy of the trained heads nor that of the transfer (GN, Hessian continuation) changes beyond the seed-to-seed spread, and the wall-clock time is the same at every . This is expected: both second-order methods depend only on the span of the basis. At the trained-head error increases by 30–40% on Burgers and Allen–Cahn, so the penalty is not free.
What changes is the basis itself. Without the penalty, (which should be 1 for a perfectly orthonormal basis) ranges from to over the heads. However, at it is within 7% of 1 for heat and between and for the other equations. At it is within 2% of 1 for all four (Fig. 12b). The second effect is uniqueness (Fig. 12a). Let be the right singular vectors of at a held-out head and , with the spread of that coordinate over the trained heads. For , the transferred head agrees with the jointly trained one () only in the first (heat) and (wave) of the 48 directions; then grows as falls and reaches order one in about 30% of the directions, while the relative error stays at or below that of the trained head. Those coordinates are not determined by the residual, and the value of the corresponding components of the weight vector is degenerate. With increasing the number of determined directions () grows for heat and for wave. Burgers and Allen–Cahn have no undetermined directions at this basis size, with or without the penalty.
The sentence on lines 255–256 ("With varying ICs, changes too and head distances measure the part of the solution carried by the head only").
We agree that the sentence is unclear and have rephrased it (Section 4.1.3). With a fixed IC and different solutions coming from variations of the EP, and
so the difference between two solutions is linear in the difference of their heads, and for an orthonormal basis is the distance between the network outputs and (Eq. 11), later weighted by the envelope . In the four multi-EP models, the rank correlation of the head distance with the distance between network outputs is , and with the distance between solutions (after reweighting) it is (heat), (wave), (Burgers) and (Allen–Cahn). Without the orthonormality penalty these drop to , , , for , and , , , for .
With varying ICs, and
The heads then describe only how each solution evolves away from its own initial profile,
and two solutions can be far apart while their heads are close. In the four multi-IC models the rank correlation of the head distance is to with the distance between network outputs, to with the distance between the evolved parts , and to with the distance between the full solutions, which is dominated by (rank correlation to ). These numbers are also reported in Appendix H.1.
L-BFGS, the condition on line 312, Adam without a time budget
L-BFGS as a baseline.
We thank the referee for this point. We have run transfer learning tests with second order optimization methods as well, mainly L-BFGS and Levenberg–Marquardt optimizers. The multi-head training of the basis with L-BFGS had been tested prior to the initial submission, but was not used moving forward because it led to overfitting of the basis to the training solutions. Also, the optimization times we are interested in are in the TL phase (which is the one that is repeated many times when looking for new solutions), in contrast to the MH training phase which is run only once per model. The comparison for the eight models of Table 1, on CPU, is reported in the new Appendix E.1 (Table 7) and summarised in Section 5.1. The table below gives the same comparison for the multi-IC models with every computation on a GPU.
Head solvers for TL to unseen in the multi-IC setting (15 targets per model, as in Table 1), all on the same frozen basis, collocation system and start head, with every computation on a GPU (NVIDIA A30): median relative error against the reference solution and median wall time (online) per transfer. Levenberg–Marquardt (LM) and L-BFGS stop with the rule of Gauss–Newton (GN). The Adam baseline is shown twice: run for the wall-clock time of the slower second-order method (the Hessian continuation), and run to a fixed budget of 2000 steps.
| PDE | GN error | GN ms | LM error | LM ms | L-BFGS error | L-BFGS ms | Hessian + RK4 error | Hessian + RK4 ms | Adam error (wall-clock of Hessian) | Adam error (2000 steps) | Adam ms (2000 steps) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| heat | 6.9 | 14 | 202 | 88 | 1647 | ||||||
| wave | 6.1 | 14 | 299 | 84 | 1856 | ||||||
| Allen–Cahn | 25 | 20 | 164 | 100 | 1861 | ||||||
| Burgers | 20 | 10 | 55 | 98 | 1797 |
On line 312, is not clear.
We thank the reviewer for pointing this out, the statement was given without explanation. The quantity is the gradient of the loss, , which is Eq. (16) written in matrix form. The Gauss–Newton step of Eq. (15) is this same gradient multiplied by a matrix: since and the columns of are linearly independent, the pseudo-inverse is , so the step is . The matrix is invertible, so the step is zero exactly when the gradient is zero. In other words, Gauss–Newton stops moving only at a stationary point of the loss, which is the condition that the Hessian continuation follows along its path. The two methods therefore look for the same kind of point and differ only in how they get there: Gauss–Newton iterates directly at the target , while the continuation starts from a trained head and follows the minimizer as changes. This does not mean that they always end at the same point; the cases where they do not are reported in Appendix B.4. The SVD cutoff used in the Gauss–Newton solve does not change this, since no singular value is discarded in our experiments. We have rewritten that sentence to include this explanation (Section 4.2.3, below Eq. 16).
Reporting in the main text the results on Adam without the same wall-clock time (Appendix B.6 and Table 3).
We have added two columns to Table 1 with Adam run without a time budget, stopped at the iteration where its residual has improved by less than 1% over 200 steps, and a summary in Section 5.1: run to convergence, Adam matches the second-order errors in the multi-EP experiments and for the heat and wave ICs, but takes roughly 15 to 390 times longer than GN, and it remains less accurate for the Allen–Cahn and Burgers ICs. The details remain in Appendix B.6 (Table 3).
Figure 3, sparser trained heads and extrapolation
In Figure 3 the known heads are already very dense. What happens with a lot of space in between, or far extrapolation?
We apologize if Figure 3 may have given the wrong impression. The continuation curve is not an interpolation of the trained heads, but is an implementation of the Hessian continuation method. As described in Section 4.2.3, starting from the head at the smallest the method first applies a Gauss–Newton solve that places this head on the minimiser of the PDE residual. It then integrates Eq. 20 with RK4 up to the largest . Each integration step evaluates the slope from the PDE residual at the current , such that the other 15 trained heads are never used and are drawn only as a check. This is now stated in the caption of Figure 3 and in the new Appendix G.
We had also tested the accuracy of this procedure with respect to the starting distance from the target without reporting it. From a head up to 7.5 grid spacings away from an unseen , Gauss–Newton and the continuation reach the same error as from the nearest head for the four equations. Adam with a fixed budget does not (wave: at 0.5 spacings, at 7.5) (Table 13).
To address sparser training directly, we retrained the four models of Figure 3 with 12, 8 and 4 heads instead of 16, keeping the eight basis functions, and evaluated them at the same 15 unseen (Table 14). With 8 heads the trained are twice as far apart. The median error is against for Burgers and unchanged for the other three equations, and the continuation started from the smallest alone still reproduces the Gauss–Newton heads over the whole range (Fig. 11a). With 4 heads, fewer than the basis size, heat and Allen–Cahn remain accurate ( and ). The wave basis however no longer represents unseen () and Burgers degrades () (Fig. 11b). This suggests using at least as many training heads as basis functions.
Extrapolation is reported in Appendix B.9 (Table 6), which we should have pointed to from the main text; Section 5.1 now does. For varying , Gauss–Newton and the continuation return the same head and the error grows with the distance from the trained range. It exceeds 10% one grid spacing out for all equations but heat. We have now separated the two ends of the range and computed the best fit that the basis allows (Table 15). Above the largest trained the error grows as a power of the distance (exponent 0.9 to 1.5) and stays within about a factor of two of that best fit, except for wave. Below the smallest trained the degradation is mild for heat and Allen–Cahn (at most at eight spacings) and stronger for Burgers. The limitation is therefore the frozen basis and not the transfer method. For varying ICs, Gauss–Newton keeps its interpolation accuracy outside the training box for all equations except Burgers.
The new Appendix G reports the start-distance and sparse-head tests and repeats Figure 3 for the models trained with 8 and 4 heads (Tables 13–15, Fig. 11).
Latent dimension of the VAE and boundary conditions
Motivation behind the choice of the latent dimension of the VAE.
The motivation is merely for visualization purposes, in the sense that one can represent/visualize the surface w(z) presented in Figs. 4 and 8 (Section 4.1.2, footnote 1). It is true that it should not be too high because one is looking for a low-dimensional parametrization of the ICs (a profile sampled on the spatial grid lives in Rnx, which becomes far too large for this purpose), but it can definitely be higher. More concretely, for a specified family of possible ICs (e.g. ones that can be produced from a finite set of K Fourier modes as in Eq. 8), a larger dimension dz of the encoding space Rdz translates in a lower loss (better training) of the VAE. This can improve accuracy of the TL results if a new instance of IC is sampled directly in Rnx and encoded into z⋆=E(IC), to later be decoded as IC=D(z⋆). In other words, higher dz is expected to produce higher agreement between IC and IC, and therefore a more accurate answer for the continuation from the learned latent space. In the current work and for our purposes, we sample z⋆ directly in the encoding space Rdz and generate new ICs from the trained decoder D(z⋆), so no real effect comes from the choice dz=2 besides the visualization of Figs. 4 and 8 and the interactive visualizations in the repository Weight_Space_Flows (https://anonymous.4open.science/r/Weight_Space_Flows).
The initial condition is a sum of sine functions which only works with null boundary conditions.
Indeed, this is a choice made for simplicity and to have a controlled experimental workspace, where we reparametrize the output of the NNs to satisfy exactly the IC and BCs. However, this reparametrization (Eq. 6, Section 4.1) is general for any time-independent boundary value problem (BVP) with IC profile A(x,0)=u0(x), not only for the cases u0(xlo)=u0(xhi)=0, so it is directly applicable in the general time-independent BVP; one would only have to choose a family of ICs that does not vanish at the boundary, and A(xlo,t)=ulo, A(xhi,t)=uhi. However, it does not cover time-dependent boundary data as it is structured now.
Concretely, one can see in Eq. 6 that, since 1−e−tp vanishes at t=0 and x~(1−x~) vanishes at xlo and xhi, the envelope g is zero at t=0, x=xlo and x=xhi, so uj(x,0)=Aj(x,0)=u0,j(x) and uj(xlo,t)=Aj(xlo,t), uj(xhi,t)=Aj(xhi,t) hold identically, whatever the head wj. For p=2 the time factor additionally satisfies ∂t(1−e−t2)t=0=0, hence gt(x,0)=0 and ∂tuj(x,0)=∂tAj(x,0)=0 is exact as well, while ∂tt(1−e−t2)t=0=2=0 leaves ∂ttuj(x,0) unconstrained, as required by the wave equation. The constraints are thus met by construction and never enter the loss.