- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a framework for evaluating whether warm starts from a learned neural PDE solver are useful. The authors use it to ask whether a more accurate prediction actually reduces the work needed to correct it, whether pretraining helps on new problems, and whether any benefit remains after accounting for the cost of the full solver.
Critical strengths
- While many works of literature assume the benefits of a warm start from a neural PDE solver, this paper challenges this assumption. I find this topic well-motivated.
- The authors show, both through a concrete example and experiments with trained models, that initial error measures do not reliably predict how much correction work a warm start will require. This is a useful challenge to evaluations that judge warm starts mainly by prediction accuracy.
Critical weaknesses
- The paper is very difficult to follow. Section 2 introduces a lot of notation that does little to help the reader understand the later results. The authors also use paper-specific terms, such as “source models,” “residual weights,” and “waves,” without defining them clearly when they first appear. Many section headings state an interpretation before explaining what was tested, and the paper gives the reader little foreshadowing for its numerous experimental comparisons. Even the three questions in the introduction are phrased so abstractly that they do not provide a clear roadmap for the paper.
- Many results are reported in terms of correction iterations. However, fewer iterations do not necessarily mean a faster solver, since neural inference and different correction steps have different costs. The authors do report wall-clock times in Section 6, but the earlier iteration results are often presented as benefits before their practical significance is clear. I would have liked the timing results to be connected more directly to the corresponding iteration results.
Questions for authors
- How sensitive are the results to the training data? The main comparisons use two source datasets, and the uncertainty intervals hold the trained models fixed. Would the reversal rates look similar across many independently drawn source datasets?
- What causes the ranking reversals in the trained models? Section 3 proves that the three error summaries are insufficient, but can the authors identify which error components make a particular learned prediction harder for CG to correct?
4 comments
Do the same test problems reverse across training draws?
Your question about independent source datasets has two parts. A similar mean reversal rate can recur in these fitted libraries while the particular tasks that reverse change from one library to the next. We used the same held-out problems across four new training draws so that we could inspect both effects. The first count asks how often a more accurate model costs CG steps on a given task. The second asks whether the same physical problems exhibit that phenomenon each time we fit a new set of models.
What varied across training draws. Each draw had 384 training problems and 64 source-validation problems. Within a draw we trained six base FNOs on the same examples, pairing three optimizer seeds across prediction loss and prediction loss with a discrete residual penalty. The positive weight was fixed at 0.01 in all four draws. Each trajectory had a 6,000-update budget, with checkpoint selection on its source-validation data. We then froze the models and ran plain CG and Poisson-preconditioned CG from their predictions on the same 64 in-distribution diffusion problems at grid size 63. All 3,072 endpoints met the true-residual acceptance test. Changing the training set under this design leaves the test operator and right-hand side fixed for each physical problem.
The denominator matters. We first require both predictions to have relative solution error at most 5%. We then retain a pair only if the same prediction has smaller solution error, energy error and residual. For each problem we divide the number of pairs for which this prediction requires more CG steps by the number of eligible pairs. We average these fractions over problems with an eligible pair. Several pairs can share a prediction and some problems have more eligible pairs than others. Counting every pair as a new independent problem would change the weighting and the meaning of the uncertainty interval.
In source pools 4, 5, 6 and 8, respectively, 80 of 372, 87 of 419, 64 of 321 and 102 of 425 eligible pairs reversed under plain CG. The task-weighted rates were 23.22%, 20.45%, 20.12% and 24.02%. In pool 6, 63 problems had an eligible pair. All 64 did so in each other pool. The original two libraries gave 23.50% and 23.87% under the same rule. Only FNO predictions qualified in those original libraries, even though they also contained CNNs. Restricting the original analysis to FNOs reproduces its per-task counts. This makes the comparison with the four new FNO-only libraries interpretable.
Which physical problems reverse? For the second question we take the intersection of the eligible tasks across the four new pools. That leaves 63 physical problems. Within each pool we mark a problem once if any eligible pair reversed. Three problems had no reversal in any pool. Eight had one or more reversals in exactly one pool, 16 in two, 22 in three and 14 in all four. Thus 60 of 63 had a reversal somewhere, yet 46 of 63 changed their yes/no outcome across the four draws. The same problem need not reverse for the same pair of checkpoints. In fact the networks are refitted on each source dataset. These are counts of affected problems, which explains why they can exceed the fraction of eligible pairs that reverse.
The two summaries answer different practical questions. If a solver group trains a new library with the same recipe, our four draws give a sense of the range observed for the overall rate. If that group wants to flag a particular target case in advance, the identities of affected tasks vary appreciably across draws. Selecting only the 14 consistently affected tasks would miss many observed reversals. Conversely, pooling the four draws and treating the 63 shared problems as four independent test sets would give an unjustified impression of precision.
The same pairs under another corrector. The initial predictions and their accuracy ordering are available before running either solver. On the same 1,537 eligible pairs in the four new pools, Poisson-PCG produced zero observed reversals. It produced 255, 272, 250 and 280 ties in those pools. A tie means the accuracy advantage bought no difference in integer iteration count. Comparing those results to plain CG keeps training and the initial prediction fixed and changes the algorithm responsible for the remaining work. The finding is specific to the preconditioner, tolerance and problems tested.
The supplement makes the cross-pool check short to reproduce. In extension_checks/source_pools/per_task.csv, select the fresh CPU FNO rows for CG and strict three-metric dominance. For each task and pool, retain positive eligibility and test whether the number of reversed pairs exceeds zero. Intersect the task IDs across pools before counting in how many pools a given problem reverses. The reported bootstrap intervals within a library resample physical problems. We keep the four source draws visible individually so that readers can see the training-data variation that motivated your question.
A control for the spectral explanation
We applied the same intervention to pairs whose more accurate member already finished earlier. This group lets us test whether the exchange can also induce the reversal seen in Figure 2. Among archive pairs with a strictly negative CG gap and a nonpositive gap on CPU replay, the exchange created a positive gap in 85 of 275 pairs in library 0 and 78 of 287 in library 1. Here the gap is the step count for the initially more accurate prediction minus the step count for its comparator. A positive gap means the initial accuracy ranking chose the slower start in iteration count.
What is exchanged? We diagonalize the actual symmetric positive definite diffusion operator for the task and split its spectrum into three equal-width intervals on the log-eigenvalue axis. We express each prediction's initial residual in that basis. Within the selected band, each recipient takes its partner's pattern of coefficients, rescaled to its own original band norm. All coefficients outside the band stay in place. Thus each recipient keeps its total initial residual norm as well as the amount of squared residual in each band. Only its distribution within the chosen band changes. The largest relative drift in the full residual norm over the population experiment was below 3.5e-14.
This construction addresses the distinction you raised between an example allowed by theory and a cause that can be examined in a trained prediction. The solution error in an eigenvector direction is proportional to the residual component divided by its eigenvalue. Moving residual within the low band can therefore make a sizeable change in solution coordinates. In the worked case from library 1, the more accurate prediction originally needed two extra steps. After the low-band exchange it finished 17 steps earlier. Its relative solution error changed from 2.41% to 2.06%, and the other prediction's from 3.49% to 4.09%. Both remained accurate and their original ordering also held in energy error and residual.
How much is numerical replay contributing? We locked all 770 eligible pairs in the two original libraries (63 x 63 grid) and replayed their starts on the same CPU used for the exchanges. All 770 pairs retained the original accuracy inequalities and the 5% cutoff. Of 173 archived positive gaps, 165 were still positive on replay, with 69 in library 0 and 96 in library 1. Those are the denominators for the removal counts. We compared each exchange to its own same-machine replay. Reconstructing the complete starting field in the eigenbasis with its original coefficients closed only six of the 165 gaps, all by producing a tie. Exact self-donation, which returns a state's own components, reproduced the unmodified solve in all 703 state controls.
How specific is the effect to the low band? A positive replay gap became zero or negative for 67 of 69 pairs in library 0 and 94 of 96 in library 1 after the low-band exchange. In 160 of those 161 pairs the originally more accurate prediction actually finished earlier. Middle-band exchanges closed 25 gaps across the two libraries and high-band exchanges closed 10. Weighting each physical task equally, the low-band exchange changed the mean gap by -13.25 and -13.32 steps in the two libraries. Performing the middle-band exchange on the same pairs changed it by -0.32 and -0.69 steps. These comparisons use the same stopping rule and the same initial pair, so the affected part of the residual is explicit.
The accuracy condition also deserves a second check after perturbation. Of the 161 pairs whose positive gap closed under low-band exchange, 92 still had both relative solution errors at most 5% and all three initial measures ordered as before. In the other 69, the change in step ordering remains measurable, but we can no longer interpret it as a comparison satisfying all the original eligibility conditions. Reporting both counts matters because holding a residual norm fixed does not hold the solution norm fixed. An exchange in a lower band also need not produce the same size displacement in solution space as an exchange in a higher band.
For anyone wishing to inspect a particular pair, Appendix X.6 and extension_checks/spectral/swaps.csv give the archived and replayed gaps, the exchanged counts, and the post-exchange accuracy checks for each band. The full study comprises 2,310 pair-and-band exchanges. Every exchanged-state solve met the prescribed true relative residual tolerance. These records make it possible to examine ties and counterexamples alongside the headline result. They support a concrete interpretation of Figure 2. Two trained predictions can start in an accuracy order on which three common summaries agree and still have different correction trajectories because of how their residual is distributed inside the low end of this operator's spectrum. Computing that eigendecomposition is an offline experiment. An affordable diagnostic for choosing a start during a solve remains an open problem.
Thank you for asking about the error the solver actually sees
Thank you for asking which components of a trained prediction actually cost the solver extra steps. Your question led us back to the same two initial fields and to the residual traces now shown in Figure 2. Our motivation is practical. Someone choosing between two networks usually has an accuracy table before they have integrated either network into a solver. They need to know when that table can point to the more expensive start. We hope the new training draws, the spectral controls, and the paired measurements of steps and time address your concerns about soundness and presentation. If the revision warrants a different judgment in your view, would you consider raising your overall rating? Thank you very much!
Changing the training data and tracing one CG reversal
Your review identified a difficulty that affected both the exposition and the scientific argument. An accuracy table tells a reader which network predicts a field more closely. It does not tell a solver developer which prediction to pass to CG, or whether calling either network will save time. We reorganized the paper around a pair of predictions for which those decisions can be followed from start to finish.
One problem, two initial fields. The opening example uses two Fourier neural operators on the same held-out diffusion problem. Their relative solution errors are 2.41% and 3.49%. The first prediction also has the smaller energy error and the smaller residual. We give CG the same matrix and right-hand side in both runs and stop at a true relative residual of 1e-8. The more accurate start takes 311 steps and its comparator takes 309. The gap is small enough that it matters how it was measured. Figure 2 shows the residual traces and their first crossings of the stopping threshold. It also follows the same pair through a change to the initial residual that increases the separation and reverses which start finishes first.
Section 2 now introduces each quantity where the solver uses it. The source model is the network fitted before the target problem arrives. Its output supplies the initial field. The residual is what remains when that field is substituted into the discrete equation. A residual weight is the coefficient on a PDE residual penalty in the training loss. The initial and expanded training runs are the two groups previously called waves. The introduction now tells the reader which comparisons test prediction rankings, which isolate the numerical corrector, and where elapsed times enter.
What we learned by changing the training data. We fitted 24 additional FNOs on four independently drawn source datasets. For each dataset we held the architecture and training budget fixed, used two losses and three optimizer seeds, and evaluated all six checkpoints on the same 64 held-out problems. Both predictions in an eligible pair had to lie within 5% relative solution error, with one strictly better on all three initial measures. Giving each eligible problem equal weight, the CG reversal rates were 23.22%, 20.45%, 20.12% and 24.02%. The original two libraries gave 23.50% and 23.87%. Four new draws support a recurring effect under this training recipe. Each interval in the manuscript still describes variation over test problems for its fitted library.
What changed the worked example. The lowest 20 eigenmodes carried less than 2% of each squared residual and more than 88% of each squared solution error. We exchanged the directions of the residual components in this band, preserving the norm of the band separately for each recipient and leaving the remaining coefficients in place. The new counts were 303 and 320. The first prediction retained its advantage on all three initial accuracy measures. Classical CG theory explains why scalar summaries can miss information. The exchange shows which part of two trained predictions can change their ranking on a given operator. It is a reference-based diagnostic that we ran offline.
How much are the steps worth? The answer is visible even within one model-selection comparison. In library 0 on in-distribution problems at grid size 63, the model chosen by validation error averaged 320.66 CG steps and 9.39 ms per query. Choosing by validation work gave 318.16 steps and 9.33 ms. Starting from zero took 333.47 steps and 7.26 ms. The learned choice improved the iteration count, and a user seeking the quickest of these three measured paths would choose zero. Each complete path was timed once per task in this experiment. We report the times descriptively.
Section 6 examines a separate workload where learned initialization did pay for itself. On eight fresh batches of 32 diffusion problems, with GPU inference and single-thread CPU correction, the physics-trained FNO reduced mean Poisson-PCG work from 60.32 to 58.27 steps. Its complete online path saved 0.201 ms per query relative to zero, or 0.73%. Two laptop sessions on the same batches and checkpoints recovered the 2.05-step saving and took 3.336 and 1.780 ms more per query than zero. The phase timings help explain the change. On the laptop, preparing inputs, transferring them and running the network cost more than the correction time saved. Hardware and software both changed between the two configurations.
These comparisons give a reader a way to use the paper when choosing a checkpoint. A prediction ranking is inexpensive to obtain on labeled validation data. Measuring a few candidate starts with the intended corrector tests whether that ranking survives correction. Timing the complete path then answers whether the saved work covers the total cost of producing the initial field. The revised paper gives the measurements for each decision and the common acceptance check that makes the comparison meaningful.