acceptodds
Under review as a conference paper at ICLR 2027

Transferability of Learned States in Neural PDE Solvers

Abstract

Assessing useful reuse in neural PDE solvers is challenging: final accuracy can reflect source learning and target-time computation. Our reuse contract separates solution accuracy, learning contribution, and numerical utility through paired state comparisons, matched target information and budgets, and cost accounting. A literature audit extracts 18 version-specific protocol records from 12 papers, documenting retained states, target-time resources, and reported controls. For a fixed linear system and residual tolerance, we construct two initial guesses with identical solution-error, energy-error, and residual norms, reaching the same solution with different conjugate-gradient (CG) iteration counts. Across 240 core source-training trajectories, two linear PDE families, Fourier neural operators and convolutional networks, a fixed predictor's benefit reverses across correction algorithms. Among pairs with both relative prediction errors on 64 in-distribution tasks ( interior grids), reductions in all three norms accompany more CG iterations, at mean taskwise rates of 23.5% and 23.9% in two libraries. Work-based selection saves 2.50–3.33 CG iterations on held-out in-distribution tasks; matched adaptation demonstrates finite-budget pretraining value. Independent batches confirm a 0.73% complete online saving for one physics-trained Fourier neural operator against zero-initialized Poisson-preconditioned CG. Reuse requires matched state comparisons and downstream computational evidence.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Physics-informed neural networks, neural operators, reusable neural solvers, scientific machine learning, transfer and adaptation, operator learning, generalization, evaluation, physical consistency, computational cost

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GXP3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a framework for evaluating whether warm starts from a learned neural PDE solver are useful. The authors use it to ask whether a more accurate prediction actually reduces the work needed to correct it, whether pretraining helps on new problems, and whether any benefit remains after accounting for the cost of the full solver.

Critical strengths
  1. While many works of literature assume the benefits of a warm start from a neural PDE solver, this paper challenges this assumption. I find this topic well-motivated.
  2. The authors show, both through a concrete example and experiments with trained models, that initial error measures do not reliably predict how much correction work a warm start will require. This is a useful challenge to evaluations that judge warm starts mainly by prediction accuracy.
Critical weaknesses
  1. The paper is very difficult to follow. Section 2 introduces a lot of notation that does little to help the reader understand the later results. The authors also use paper-specific terms, such as “source models,” “residual weights,” and “waves,” without defining them clearly when they first appear. Many section headings state an interpretation before explaining what was tested, and the paper gives the reader little foreshadowing for its numerous experimental comparisons. Even the three questions in the introduction are phrased so abstractly that they do not provide a clear roadmap for the paper.
  2. Many results are reported in terms of correction iterations. However, fewer iterations do not necessarily mean a faster solver, since neural inference and different correction steps have different costs. The authors do report wall-clock times in Section 6, but the earlier iteration results are often presented as benefits before their practical significance is clear. I would have liked the timing results to be connected more directly to the corresponding iteration results.
Questions for authors
  1. How sensitive are the results to the training data? The main comparisons use two source datasets, and the uncertainty intervals hold the trained models fixed. Would the reversal rates look similar across many independently drawn source datasets?
  2. What causes the ranking reversals in the trained models? Section 3 proves that the three error summaries are insufficient, but can the authors identify which error components make a particular learned prediction harder for CG to correct?
4 comments
AuthorsOpenReview ↗

Changing the training data and tracing one CG reversal

Your review identified a difficulty that affected both the exposition and the scientific argument. An accuracy table tells a reader which network predicts a field more closely. It does not tell a solver developer which prediction to pass to CG, or whether calling either network will save time. We reorganized the paper around a pair of predictions for which those decisions can be followed from start to finish.

One problem, two initial fields. The opening example uses two Fourier neural operators on the same held-out diffusion problem. Their relative solution errors are 2.41% and 3.49%. The first prediction also has the smaller energy error and the smaller residual. We give CG the same matrix and right-hand side in both runs and stop at a true relative residual of 1e-8. The more accurate start takes 311 steps and its comparator takes 309. The gap is small enough that it matters how it was measured. Figure 2 shows the residual traces and their first crossings of the stopping threshold. It also follows the same pair through a change to the initial residual that increases the separation and reverses which start finishes first.

Section 2 now introduces each quantity where the solver uses it. The source model is the network fitted before the target problem arrives. Its output supplies the initial field. The residual is what remains when that field is substituted into the discrete equation. A residual weight is the coefficient on a PDE residual penalty in the training loss. The initial and expanded training runs are the two groups previously called waves. The introduction now tells the reader which comparisons test prediction rankings, which isolate the numerical corrector, and where elapsed times enter.

What we learned by changing the training data. We fitted 24 additional FNOs on four independently drawn source datasets. For each dataset we held the architecture and training budget fixed, used two losses and three optimizer seeds, and evaluated all six checkpoints on the same 64 held-out problems. Both predictions in an eligible pair had to lie within 5% relative solution error, with one strictly better on all three initial measures. Giving each eligible problem equal weight, the CG reversal rates were 23.22%, 20.45%, 20.12% and 24.02%. The original two libraries gave 23.50% and 23.87%. Four new draws support a recurring effect under this training recipe. Each interval in the manuscript still describes variation over test problems for its fitted library.

What changed the worked example. The lowest 20 eigenmodes carried less than 2% of each squared residual and more than 88% of each squared solution error. We exchanged the directions of the residual components in this band, preserving the norm of the band separately for each recipient and leaving the remaining coefficients in place. The new counts were 303 and 320. The first prediction retained its advantage on all three initial accuracy measures. Classical CG theory explains why scalar summaries can miss information. The exchange shows which part of two trained predictions can change their ranking on a given operator. It is a reference-based diagnostic that we ran offline.

How much are the steps worth? The answer is visible even within one model-selection comparison. In library 0 on in-distribution problems at grid size 63, the model chosen by validation error averaged 320.66 CG steps and 9.39 ms per query. Choosing by validation work gave 318.16 steps and 9.33 ms. Starting from zero took 333.47 steps and 7.26 ms. The learned choice improved the iteration count, and a user seeking the quickest of these three measured paths would choose zero. Each complete path was timed once per task in this experiment. We report the times descriptively.

Section 6 examines a separate workload where learned initialization did pay for itself. On eight fresh batches of 32 diffusion problems, with GPU inference and single-thread CPU correction, the physics-trained FNO reduced mean Poisson-PCG work from 60.32 to 58.27 steps. Its complete online path saved 0.201 ms per query relative to zero, or 0.73%. Two laptop sessions on the same batches and checkpoints recovered the 2.05-step saving and took 3.336 and 1.780 ms more per query than zero. The phase timings help explain the change. On the laptop, preparing inputs, transferring them and running the network cost more than the correction time saved. Hardware and software both changed between the two configurations.

These comparisons give a reader a way to use the paper when choosing a checkpoint. A prediction ranking is inexpensive to obtain on labeled validation data. Measuring a few candidate starts with the intended corrector tests whether that ranking survives correction. Timing the complete path then answers whether the saved work covers the total cost of producing the initial field. The revised paper gives the measurements for each decision and the common acceptance check that makes the comparison meaningful.

AuthorsOpenReview ↗

Do the same test problems reverse across training draws?

Your question about independent source datasets has two parts. A similar mean reversal rate can recur in these fitted libraries while the particular tasks that reverse change from one library to the next. We used the same held-out problems across four new training draws so that we could inspect both effects. The first count asks how often a more accurate model costs CG steps on a given task. The second asks whether the same physical problems exhibit that phenomenon each time we fit a new set of models.

What varied across training draws. Each draw had 384 training problems and 64 source-validation problems. Within a draw we trained six base FNOs on the same examples, pairing three optimizer seeds across prediction loss and prediction loss with a discrete residual penalty. The positive weight was fixed at 0.01 in all four draws. Each trajectory had a 6,000-update budget, with checkpoint selection on its source-validation data. We then froze the models and ran plain CG and Poisson-preconditioned CG from their predictions on the same 64 in-distribution diffusion problems at grid size 63. All 3,072 endpoints met the true-residual acceptance test. Changing the training set under this design leaves the test operator and right-hand side fixed for each physical problem.

The denominator matters. We first require both predictions to have relative solution error at most 5%. We then retain a pair only if the same prediction has smaller solution error, energy error and residual. For each problem we divide the number of pairs for which this prediction requires more CG steps by the number of eligible pairs. We average these fractions over problems with an eligible pair. Several pairs can share a prediction and some problems have more eligible pairs than others. Counting every pair as a new independent problem would change the weighting and the meaning of the uncertainty interval.

In source pools 4, 5, 6 and 8, respectively, 80 of 372, 87 of 419, 64 of 321 and 102 of 425 eligible pairs reversed under plain CG. The task-weighted rates were 23.22%, 20.45%, 20.12% and 24.02%. In pool 6, 63 problems had an eligible pair. All 64 did so in each other pool. The original two libraries gave 23.50% and 23.87% under the same rule. Only FNO predictions qualified in those original libraries, even though they also contained CNNs. Restricting the original analysis to FNOs reproduces its per-task counts. This makes the comparison with the four new FNO-only libraries interpretable.

Which physical problems reverse? For the second question we take the intersection of the eligible tasks across the four new pools. That leaves 63 physical problems. Within each pool we mark a problem once if any eligible pair reversed. Three problems had no reversal in any pool. Eight had one or more reversals in exactly one pool, 16 in two, 22 in three and 14 in all four. Thus 60 of 63 had a reversal somewhere, yet 46 of 63 changed their yes/no outcome across the four draws. The same problem need not reverse for the same pair of checkpoints. In fact the networks are refitted on each source dataset. These are counts of affected problems, which explains why they can exceed the fraction of eligible pairs that reverse.

The two summaries answer different practical questions. If a solver group trains a new library with the same recipe, our four draws give a sense of the range observed for the overall rate. If that group wants to flag a particular target case in advance, the identities of affected tasks vary appreciably across draws. Selecting only the 14 consistently affected tasks would miss many observed reversals. Conversely, pooling the four draws and treating the 63 shared problems as four independent test sets would give an unjustified impression of precision.

The same pairs under another corrector. The initial predictions and their accuracy ordering are available before running either solver. On the same 1,537 eligible pairs in the four new pools, Poisson-PCG produced zero observed reversals. It produced 255, 272, 250 and 280 ties in those pools. A tie means the accuracy advantage bought no difference in integer iteration count. Comparing those results to plain CG keeps training and the initial prediction fixed and changes the algorithm responsible for the remaining work. The finding is specific to the preconditioner, tolerance and problems tested.

The supplement makes the cross-pool check short to reproduce. In extension_checks/source_pools/per_task.csv, select the fresh CPU FNO rows for CG and strict three-metric dominance. For each task and pool, retain positive eligibility and test whether the number of reversed pairs exceeds zero. Intersect the task IDs across pools before counting in how many pools a given problem reverses. The reported bootstrap intervals within a library resample physical problems. We keep the four source draws visible individually so that readers can see the training-data variation that motivated your question.

AuthorsOpenReview ↗

A control for the spectral explanation

We applied the same intervention to pairs whose more accurate member already finished earlier. This group lets us test whether the exchange can also induce the reversal seen in Figure 2. Among archive pairs with a strictly negative CG gap and a nonpositive gap on CPU replay, the exchange created a positive gap in 85 of 275 pairs in library 0 and 78 of 287 in library 1. Here the gap is the step count for the initially more accurate prediction minus the step count for its comparator. A positive gap means the initial accuracy ranking chose the slower start in iteration count.

What is exchanged? We diagonalize the actual symmetric positive definite diffusion operator for the task and split its spectrum into three equal-width intervals on the log-eigenvalue axis. We express each prediction's initial residual in that basis. Within the selected band, each recipient takes its partner's pattern of coefficients, rescaled to its own original band norm. All coefficients outside the band stay in place. Thus each recipient keeps its total initial residual norm as well as the amount of squared residual in each band. Only its distribution within the chosen band changes. The largest relative drift in the full residual norm over the population experiment was below 3.5e-14.

This construction addresses the distinction you raised between an example allowed by theory and a cause that can be examined in a trained prediction. The solution error in an eigenvector direction is proportional to the residual component divided by its eigenvalue. Moving residual within the low band can therefore make a sizeable change in solution coordinates. In the worked case from library 1, the more accurate prediction originally needed two extra steps. After the low-band exchange it finished 17 steps earlier. Its relative solution error changed from 2.41% to 2.06%, and the other prediction's from 3.49% to 4.09%. Both remained accurate and their original ordering also held in energy error and residual.

How much is numerical replay contributing? We locked all 770 eligible pairs in the two original libraries (63 x 63 grid) and replayed their starts on the same CPU used for the exchanges. All 770 pairs retained the original accuracy inequalities and the 5% cutoff. Of 173 archived positive gaps, 165 were still positive on replay, with 69 in library 0 and 96 in library 1. Those are the denominators for the removal counts. We compared each exchange to its own same-machine replay. Reconstructing the complete starting field in the eigenbasis with its original coefficients closed only six of the 165 gaps, all by producing a tie. Exact self-donation, which returns a state's own components, reproduced the unmodified solve in all 703 state controls.

How specific is the effect to the low band? A positive replay gap became zero or negative for 67 of 69 pairs in library 0 and 94 of 96 in library 1 after the low-band exchange. In 160 of those 161 pairs the originally more accurate prediction actually finished earlier. Middle-band exchanges closed 25 gaps across the two libraries and high-band exchanges closed 10. Weighting each physical task equally, the low-band exchange changed the mean gap by -13.25 and -13.32 steps in the two libraries. Performing the middle-band exchange on the same pairs changed it by -0.32 and -0.69 steps. These comparisons use the same stopping rule and the same initial pair, so the affected part of the residual is explicit.

The accuracy condition also deserves a second check after perturbation. Of the 161 pairs whose positive gap closed under low-band exchange, 92 still had both relative solution errors at most 5% and all three initial measures ordered as before. In the other 69, the change in step ordering remains measurable, but we can no longer interpret it as a comparison satisfying all the original eligibility conditions. Reporting both counts matters because holding a residual norm fixed does not hold the solution norm fixed. An exchange in a lower band also need not produce the same size displacement in solution space as an exchange in a higher band.

For anyone wishing to inspect a particular pair, Appendix X.6 and extension_checks/spectral/swaps.csv give the archived and replayed gaps, the exchanged counts, and the post-exchange accuracy checks for each band. The full study comprises 2,310 pair-and-band exchanges. Every exchanged-state solve met the prescribed true relative residual tolerance. These records make it possible to examine ties and counterexamples alongside the headline result. They support a concrete interpretation of Figure 2. Two trained predictions can start in an accuracy order on which three common summaries agree and still have different correction trajectories because of how their residual is distributed inside the low end of this operator's spectrum. Computing that eigendecomposition is an offline experiment. An affordable diagnostic for choosing a start during a solve remains an open problem.

AuthorsOpenReview ↗

Thank you for asking about the error the solver actually sees

Thank you for asking which components of a trained prediction actually cost the solver extra steps. Your question led us back to the same two initial fields and to the residual traces now shown in Figure 2. Our motivation is practical. Someone choosing between two networks usually has an accuracy table before they have integrated either network into a solver. They need to know when that table can point to the more expensive start. We hope the new training draws, the spectral controls, and the paired measurements of steps and time address your concerns about soundness and presentation. If the revision warrants a different judgment in your view, would you consider raising your overall rating? Thank you very much!

Reviewer_vcmuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that the value of a learned state in a neural PDE solver (pretrained weights, an anchor, a solver program) must be established by three separate comparisons: output correctness under a declared acceptance rule, a paired contrast against a source-independent control with matched target information and budget, and a complete-cost comparison against competent classical alternatives. It supports this with a literature audit of 18 protocol records from 12 papers, an exact 4 x 4 conjugate-gradient construction in which two initial errors with identical solution, energy, and residual norms need 2 versus 3 iterations (and a scaled version that is strictly better in all three norms yet still needs 3), and a large controlled study of FNO/CNN warm starts for 2D variable-coefficient diffusion and advection–diffusion with several correctors, a frozen-library model-selection study, matched adaptation, a batch-timing confirmation, reduced-budget audits of the official HINTS and NOWS code, and a 1D semilinear Newton extension. The headline findings are that a fixed predictor's warm-start benefit changes sign across correctors, that predictors better in all three norms can need more CG iterations, that selecting models on validation work rather than prediction error saves a few iterations, and that one physics-trained FNO yields a 0.73% online wall-clock saving.

Critical strengths

The experiment arrangement is well above the norm for this literature: paired task-level contrasts, failures retained in denominators, exploratory and confirmatory cohorts kept separate, selection choices locked before test generation, and cost boundaries that charge assembly, transfers, setup, discarded probes and process startup. The mathematics is correct; I reproduced Proposition 1 and Corollary 2 in exact arithmetic. The paper is candid about null results and ships an executable supplement with checkpoints and raw records. Fair baselines and full cost accounting for neural warm starts is a legitimate concern.

Critical weaknesses
  1. The theoretical core is a textbook property of CG restated as a proposition. Proposition 1 takes A=diag(1,2,3,4)A=\mathrm{diag}(1,2,3,4) with one initial error supported on two eigenvectors and another on three; exact-arithmetic CG terminates in 2 versus 3 steps because the degree of the minimal polynomial of e0e_0 equals the number of distinct eigenvalues carrying nonzero weight. This is the finite-termination property from Hestenes & Stiefel (1952). Eq. (4) is the standard polynomial characterization of the CG error, and the fact that Krylov convergence is governed by the full spectral distribution of e0e_0 rather than by a few of its moments (here ∑ci2\sum c_i^2, ∑λici2\sum \lambda_i c_i^2, ∑λi2ci2\sum \lambda_i^2 c_i^2) is the central thesis of Carson, Liesen & Strakoš (2024), which the paper cites as background rather than as the source of its own claim. Corollary 2 is scale invariance of CG. The "constructive diagnostic" of Appendix H uses the reference solution and a full eigendecomposition to place all error mass on at most three spectral atoms, so the near-termination it reports is built in. Attaching an exact-arithmetic verification script to a known fact does not make the fact a contribution.

  2. The "reuse contract" is standard experimental methodology under new names. Correctness is checking the output; learning contribution is an ablation against a matched control; numerical utility is an end-to-end cost comparison against a competent baseline. The notation C=(Q,I,A,B,e,V)C=(Q,I,A,B,e,V) and Δe(B;ϕ,ϕ0∣C)\Delta e(B;\phi,\phi_0\mid C) in Eqs. (1)–(2) is introduced and then essentially never used. The critique that this literature reports iteration counts instead of wall-clock, uses weak baselines, and omits classical reuse is already made by McGreivy & Hakim (2024) and Grossmann et al. (2024). The 18-record literature audit (Table 11) is a set of annotated reading notes whose "contract interpretation" column amounts to "count the target-time resources"; it produces no synthesis, taxonomy or quantitative finding about the field. The concluding sentence ("the value of learning is established by what changes when its retained state is replaced") is the definition of an ablation.

  3. The empirical findings reproduce what the cited precedents already established, and the paper concedes as much. That warm starts at equal distance from the solution converge differently, and that a warm start should therefore be trained against the downstream solver, is the premise of Sambharya et al. (2024). That one-time neural initialization is weak compared with repeated correction, and that networks leave high-frequency error that residual-based correctors see as large residuals, is the mechanism HINTS (Zhang et al., 2024) is built on; the paper's own Table 11 (record P04) writes "spectral complementarity is established prior work", and Appendix T states that the prior work "preclude[s] claiming that prediction–solver mismatch or residual feedback is new". Learned Poisson-CG initialization inside a solver loop is Um et al. (2020). Given these, the corrector-dependent sign of the warm-start benefit in Figure 1 is the expected consequence of a prediction whose residual is 7.4 times that of the zero start, and the ranking mismatch among accurate predictors is the expected consequence of integer iteration counts near 300 being set by the spectrum of AA rather than by three scalar summaries. The HINTS and NOWS audits, run at reduced budgets (NOWS at grid 64 instead of 256, 256 instead of 1000 samples, 100 instead of 2000 epochs), confirm what those papers reported and overturn nothing.

  4. There is no constructive contribution. The paper proposes no training objective, no selection policy with meaningful gains, no cost predictor and no benchmark. Work-based model selection is the obvious thing to do once a corrector is fixed, and it saves on the order of 1% of CG iterations in distribution and nothing under shift. The one positive wall-clock result is a 0.73% saving on one configuration, one seed and one node, with GPU inference paired against single-thread CPU correction. The obvious control for the flagship reversal, one or two smoothing sweeps on the prediction before CG, is never run; the paper's own residual-based selector always chooses zero initialization, which points the same way. The directions that would constitute a contribution (solver-aware training under the contract, reference-free prediction of correction cost, lifecycle optimization) are listed as future work in Appendix V.1. A diagnostic paper can merit acceptance when the diagnosis changes practice or reverses a published claim; this one does neither.

  5. Presentation and framing. The main text is a sequence of numbers with stacked caveats; terms such as "retained state", "native-interface check" and "matched target information" are never pinned down with a running example; Figure 2 is cited in Section 5.3 but lives in Appendix G; the submission is 59 pages with 39 tables. Phrases such as "revision-stage experiment", "historical aggregate blocks", "retained from the source manuscript", "recovered checkpoints" and "revises an earlier negative result" refer to a prior review round the reader has not seen and should not appear in a double-blind submission. Appendix W states that training arrays, optimizer trajectories and native-method checkpoints are not included, so the supplement supports reanalysis of supplied records but not replication.

Questions for authors
  1. State precisely which result in the paper is not already contained in the finite-termination property of CG, Carson, Liesen & Strakoš (2024), Sambharya et al. (2024), or Zhang et al. (2024).

  2. Does applying the contract reverse any conclusion of a published paper? For instance, do HINTS or NOWS claims change when audited at their native budgets with the paper's cost accounting?

  3. Do one or two Jacobi or Gauss–Seidel sweeps on the FNO prediction before CG remove the reversal in Figure 1?

  4. Would a solver-aware training objective in the style of Sambharya et al. (2024), evaluated under the same contract, produce a learned state whose benefit does not reverse across correctors? If so, why was it not included?

  5. How many equation/grid/batch/model conditions were examined before the n=127n=127, batch-32, physics-FNO condition was selected for confirmation?

4 comments
AuthorsOpenReview ↗

What the trained-model comparisons establish, and the controls you suggested

Thank you for reproducing the CG construction. We agree that finite termination and the polynomial description of CG are classical. Our example gives a reader a small system on which three familiar accuracy measures agree and the iteration counts differ. The empirical question starts where that example ends. Given two trained predictors that both approximate the solution well, how often would all three measures point to the one that costs more to correct? Can we change that ordering by modifying a specific part of the learned predictions while keeping the operator, right-hand side and stopping rule fixed?

How often does this matter on actual predictions? On 64 held-out diffusion problems we retained pairs with relative solution errors at most 5% and a strict ordering in solution error, energy error and residual. The more accurate prediction took more plain-CG steps in 72 of 362 eligible pairs in one library and 101 of 408 in the other. Different problems supply different numbers of pairs. Giving each physical problem equal weight yields rates of 23.50% and 23.87%. A reversal occurred on 42 and 47 of the 64 problems, respectively. Conditional on a reversal, the extra work averaged about four steps and ranged from one to thirteen across pairs (Table 39). This is a measurable cost for choosing between good models, even though the typical difference is modest. All eligible pairs here came from Fourier neural operators.

Which part of a trained state changes the result? We replayed the 770 eligible pairs and kept the intervention denominator explicit. Of 173 archived reversals, 165 still had a positive iteration gap on CPU. We exchanged residual directions between the two predictions within one spectral band, keeping each recipient's norm in that band and its other coefficients fixed. The low-band exchange removed 161 of the 165 gaps, counting ties. The middle and high bands removed 25 and 10. In 92 of the 161 cases, both predictions remained within 5% and the original ordering of all three accuracy measures survived. Full-spectrum reconstruction alone turned six gaps into ties. The low band can also move the ordering the other way. Among 562 controls with a negative archived gap and a nonpositive replay gap, it created 163 positive gaps. This contrast isolates a feature of the trained residuals on the actual test matrices. The experiment uses an offline eigendecomposition and reference solution (Figure 2 and Appendix X.6).

Does smoothing remove the effect? Your Jacobi control gives this question a direct test. On the 16 joint-shift problems, giving only the FNO two damped sweeps still left it 6.19 and 6.56 plain-CG steps behind unsmoothed zero in the two libraries. With both starts smoothed twice, the same FNO took 7.62 and 8.00 more plain-CG steps but saved 1.90 and 2.06 Poisson-PCG steps. One sweep gave the same signs. All 672 solves passed the true residual check. A short smoothing pass therefore leaves an observable dependence on which corrector receives the field.

What happens after training through the solver? Following the motivation in Sambharya et al., we continued six pretrained FNOs for 300 updates. One arm scored the raw output and one scored the result after five differentiable ordinary-CG steps. The models started from identical weights within each comparison and saw the same minibatches. Their training compute differed because the second loss includes those steps. On the held-out shift, the solver-aware models took 8.67 and 8.79 more plain-CG steps than zero, yet saved 1.08 and 1.21 Poisson-PCG steps. Compared directly with matched data continuation, the solver-aware arm took 0.50 additional CG steps in the first library and 0.10 in the second. The paired interval for the second difference spans zero (Table 45). A different training horizon or an objective targeting PCG is an open test.

How did the timing comparison enter the paper? We screened 48 combinations of equation, grid, batch size and learned start, using two exploratory groups per condition. We then fixed diffusion, grid 127, batches of 32 and a single physics-trained FNO checkpoint before running eight fresh groups. The discovery groups were excluded. At the same verified tolerance the model saved 0.201 ms per query on the original server, with a paired 95% interval of 0.171 to 0.230 ms. The repeat sessions on a second system recovered the 2.05-step saving but took longer than zero initialization. Section 6 reports which part of the outcome reproduced and where we measured a wall-clock saving for this model and workload. Our comparison of HINTS and NOWS is bounded by the controls and training budgets discussed in our separate reply below.

The revision starts with one pair of predictions and follows it through the stopping test and the spectral exchange. This gives the reader a way to check our empirical claim without adopting notation for a general reuse protocol or treating the construction as a novel theorem.

AuthorsOpenReview ↗

A reference-free choice that changes once its cost is counted

Your question about a usable choice of initial guess gets at a distinction that matters in practice. A solver can choose a start before each query by probing several candidates, or a developer can choose one checkpoint on validation problems and deploy it on later queries. Both decisions can use the measured behavior of the corrector. They spend that information-gathering budget at different times. We evaluated both, with the classical starts included wherever a user could actually choose them.

What can we decide during a solve? In Table 16 the candidates are zero, a coarse-grid start and a data-trained FNO. At grid 63 each candidate receives five steps. The solver then restarts from the selected field. This implementation carries the field forward and discards the earlier Krylov directions. The choice uses quantities available during the solve. We timed candidate construction, inference, the three probes, the restarted solve and the final residual check. This makes it possible to evaluate the rule on exactly the information and work it would consume in use.

On 32 diffusion problems the chosen field needed 49.59 further updates on average. Zero needed 60.38. If we counted from the point of selection, the probe would appear to save 10.79 updates. Each query had already paid 15 updates across the three candidates. The complete policy took 64.59 updates and 15.30 ms per query, versus 60.38 updates and 10.22 ms for zero. All 32 problems passed the stopping test. Even choosing the smallest initial residual, without any probe steps, took 14.25 ms because the policy built the candidate fields before choosing zero. The coarse start on its own took 54.53 updates and 11.26 ms. These two rows show exactly how much the initial solve cost relative to the iterations it saved.

The same accounting can be checked independently on the advection-diffusion tasks from this experiment. The probe reduced subsequent work from 91.75 to 86.34 updates, then raised the full count to 101.34 once the initial 15 were included. Mean latency was 45.54 ms for the policy and 39.84 ms from zero. All 32 tasks passed here as well. The two families therefore give a consistent practical answer for this implementation. A policy that retained a useful Krylov basis or estimated work from fewer probes could have different costs. The zero start gives a baseline that requires no candidate search. A new policy can be tested against it on the same tasks.

What can we choose once, before deployment? We also fixed two libraries of 12 trained models and selected either by mean validation solution error or by the number of correction steps. Each policy saw the same 32 validation tasks per distribution and grid. The work rule first minimized failed solves, then mean penalized updates, with a fixed tie rule. We recorded the chosen checkpoints before generating test cases. Each library/corrector comparison required 384 candidate corrections on validation data. Their incremental CPU cost ranged from 0.19 to 4.97 seconds per configuration. The selected checkpoint can then be used directly on subsequent queries without probing the other candidates.

At grid 63 on 64 held-out in-distribution diffusion problems, ordinary CG needed 320.656 updates for the accuracy-selected state and 318.156 for the work-selected state in library 0. In library 1 the corresponding means were 320.891 and 317.562. The paired savings were 2.50 and 3.33 steps, with task-bootstrap 95% intervals of [1.05, 3.84] and [1.53, 5.02]. Under joint shift the two policies chose the same state in library 0 for that corrector and grid. In library 1 the estimated saving was 0.55 steps with an interval spanning zero. Tables 4 and 38 give the absolute counts and differences. The in-distribution savings are improvements over an explicit accuracy-based selection policy, conditional on the fitted library and validation draw.

For readers checking the comparison, acceptance required a finite solution and a recomputed relative residual at the specified tolerance. Failed runs would have received a score of 1201, one more than the cap of 1200. All 19,968 validation and test endpoints passed in the frozen-library study. Every model had to meet the same residual criterion, so a policy could earn a lower count only by reaching this criterion in fewer steps on average.

The practical recommendation follows from the difference between these two decisions. A user who has a fixed set of checkpoints and representative validation problems can measure how checkpoint choice changes downstream work, then decide whether the gain is worth the one-off validation cost. A user making that decision separately on every query must charge the discarded candidates each time. We would also measure elapsed time in the intended deployment before choosing a production path. The selection experiment recorded one time per task and path, so it supports a stronger inference about accepted update counts than about a timing optimum.

AuthorsOpenReview ↗

What the full HINTS comparison says about one-time initialization

Your question about HINTS and NOWS calls for comparisons at the level of the published methods. The two audits support different conclusions. For HINTS we had the official Poisson test set and pretrained weights, which allowed us to distinguish what an initial neural prediction does from what continued neural correction does. For NOWS we had the official implementation and retained its variational training objective, but reduced the training budget. We can therefore answer the HINTS mechanism question directly and specify which NOWS comparisons remain conditional on our training setup.

Does one neural initialization help Jacobi? Yes, when the cap allows the baseline to finish. We ran the official one-dimensional HINTS Poisson example with its original weights, Jacobi relaxation of 2/3 and one neural correction every 25 updates in the hybrid solver. All paths used a relative residual tolerance of 10^-8. Within 400 updates, repeated HINTS solved all 750 cases, whereas both Jacobi alone and a single neural start followed by Jacobi solved none. Extending the cap to 10,000 let all three methods reach the same tolerance on all 750 cases. Ordinary Jacobi then took 4,992.71 updates on average, versus 3,899.80 after a single neural initialization. The single neural call thus provided a substantial benefit even though the shorter-cap success counts could not distinguish these two paths.

Repeated HINTS took 138.84 updates on the same cases. This comprises 133.51 numerical updates and 5.33 neural calls on average. Treating one neural call as having the same cost as one Jacobi update would obscure what the hybrid actually spends. We timed a fixed 32-case subset in three randomized blocks with the model resident in memory. Each path assembled the same matrix and verified the true residual. The median times were 18.809 ms for Jacobi, 14.339 ms for one neural start, and 1.010 ms for repeated HINTS. The tridiagonal direct solve took 0.0299 ms on these systems with 29 interior unknowns. We excluded offline training and model startup from this comparison and used the same warm-model boundary for the neural alternatives (Appendix Q).

A further check in Appendix S asks what happens when the prescribed endpoint values change. On the first 32 official tasks we kept the coefficients, forcing and grid fixed and evaluated four boundary assignments. We imposed the boundary values by subtracting an affine lift, including the discrete operator applied to that lift in the corrected right-hand side. This adjustment matters for a variable coefficient. A direct solve of the full system agreed with the lifted reference to a maximum relative difference of 2.731 x 10^-14. With the same weights and correction schedule, mean HINTS updates ranged from 137.44 to 146.09 across the four assignments. All 384 method endpoints passed, including the boundary check. These are paired variants of the same 32 problems. The evidence tells us that an explicit numerical transformation lets the existing model be reused on these boundary data. Attribution to learned boundary generalization would require its own control.

How much can the NOWS audit settle? We used the official Darcy code, including its four-block FNO, coefficient preprocessing, homogeneous Dirichlet transform and VINO energy objective. Three training seeds shared one set of 256 source examples and ran for 100 epochs. We fixed the final epoch before testing on 48 physical tasks, with 16 each in distribution, under a change in coefficient contrast and under a change in roughness. At grid 128 and tolerance 10^-8, learned initialization saved 20.15 CG updates relative to zero in distribution and 15.46 under roughness shift. Under contrast shift it added 38.71. The larger grid evaluates a resampling of the same base field, so the grid comparison also has a specific interpretation.

What changing the baseline buys a reader is particularly clear for NOWS. A coarse start required fewer iterations than the neural start in all nine combinations of corrector and distribution at this grid and tolerance. Yet the coarse path built and solved an additional half-resolution problem. Charging that work made learned initialization about 0.300 to 0.330 seconds faster than coarse in all twelve timing subgroups. Against zero or direct solution, none of these subgroups established a paired time saving for the learned path. The timed implementation spent roughly 1.27 to 1.30 seconds on matrix assembly alone. Appendix R separates timings by host, takes repetition medians before averaging seeds, and reports the physical task counts. This explains how a trained predictor can be a useful alternative to a coarse solve even when its advantage over the simpler zero start remains unestablished.

For NOWS, a matched test at the native training budget would let us compare how much work each start saves at both budgets on the same tasks, with the corrector held fixed. That comparison remains open in the present study.

AuthorsOpenReview ↗

Thank you for checking the construction and the controls

Thank you for independently checking the exact CG construction and asking what a practitioner gains beyond that classical result. Your suggestion to smooth the predictions gave us a concrete way to challenge the main finding. We hope the comparison across correctors, the measured cost of selecting a start, and the tests on trained residuals address the concerns behind your initial assessment. If those results have persuaded you that the empirical contribution merits a different judgment, we would appreciate your raising the overall rating. Thank you very much!

Reviewer_w9dyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate whether accurate neural network predictions effectively reduce the work required by iterative solvers (like CG or GMRES) to reach a solution; similar to classical preconditioning. The main contribution is a conceptual framework to separate solution accuracy, learning contribution, and numerical utility. Literature review and empirical evidence show that initial error norms are insufficient to predict correction work, with frequent "ranking reversals" where more accurate predictors require more iterations. the authors also evaluate the impact of the choice of corrector and the importance of accounting for total system costs.

Critical strengths
  • The inclusion of a formal proposition and corollary regarding CG convergence provides a mathematical argument for why initial norms cannot determine total work.
  • The numerical study is fairly extensive (several parameters of the optimizers varied), even though it is done for very specific problems.
Critical weaknesses
  • The experiments focus primarily on linear diffusion and advection-diffusion; the findings for more complex or non-linear systems remain less extensive.
  • The reported "real online gain" is very small (below 1 %), which, while honest, suggests that the practical utility of the discussed methods may be limited to the specific tested scenarios.
  • the paper is very hard to read with critical definitions missing (many symbols in formulas are not explained in the text).
Questions for authors
  • On the small online gain: how sensitive is this result to different hardware configurations or software implementations of the Poisson solver?
  • Can the authors elaborate on why the "physics-trained" FNO showed a gain while the "data-trained" FNO did not ( in the online batch confirmation)?
4 comments
AuthorsOpenReview ↗

When do the saved PCG iterations pay for inference?

The physics-trained Fourier neural operator (FNO) saved 2.05 Poisson-preconditioned CG steps per query on both machines. Those steps paid for inference on the server. They also saved correction time on the laptop, but supplying the prediction there cost more than the time saved. Your two questions helped us separate what the training objective changed in the solver from what the execution environment changed on the clock.

Why did one FNO pay for itself? We compared two initial guesses for the same 256 diffusion problems. Both models were trained against reference fields. The physics-trained model also received a penalty on the squared relative residual of the discrete equation. Once a model supplies its initial guess, both paths use the same Poisson preconditioner, the same CG routine and the same 1e-8 test on the true relative residual. Starting from zero took 60.32 steps per query. Starting from the data-trained FNO took 59.41, and starting from the physics-trained FNO took 58.27. The extra reduction of 1.14 steps over the data-trained checkpoint matters when inference and correction are close to break-even. It is a measured difference between these two trained states. Establishing that the penalty causes a particular speedup across training seeds would require timing additional independently trained states.

How large is the server effect? On eight fresh batches of 32, the mean times were 27.469 ms per query from zero, 27.746 from the data-trained FNO, and 27.267 from the physics-trained FNO. Comparing within batch, the physics-trained checkpoint saved 0.201 ms (0.73%) against zero, with a paired 95% interval of 0.171 to 0.230 ms. It finished earlier in every batch. The data-trained checkpoint added 0.278 ms, with an interval of 0.241 to 0.316 ms. Thus the smaller step saving of that checkpoint still had value in PCG, while only the physics-trained checkpoint improved the complete server path. The distinction was easy to miss when our paper gave the percentage without the underlying work and times. Section 6 now puts the measurements side by side.

What reproduced on another machine? We kept the weights, physical problems, grouping and tolerance fixed. Each machine used one GPU for inference and one CPU thread for numerical correction. On the laptop, zero and physics-FNO again averaged 60.32 and 58.27 steps. In the first session the physics-FNO path took 41.534 ms per query against 38.198 from zero. In the second it took 36.956 against 35.176. The paired extra costs were 3.336 and 1.780 ms, respectively. Both paired intervals exclude zero. These sessions reuse the 256 tasks. They test repeat timing on one more system and do not add independent problems to the server confirmation. The matched inputs make the stable work saving visible across configurations.

Where did the time go? A separate instrumented run on the laptop divided the work into stages. Preparing inputs, transferring tensors, running inference and synchronizing took 5.19 ms per query for the physics-FNO initial guess, against 0.03 ms for zero. Correction and verification then took 33.57 ms against 35.09. The learned guess recovered 1.52 ms in the part of the solver it was intended to accelerate. Its additional initialization cost was 5.16 ms. For this path to win, the correction saving must cover that extra cost. We report these stage measurements separately because introducing synchronization for profiling changes the timing boundary. The two full-session comparisons above use the original timer. The archived server measurements record total times and iteration counts, so we cannot assign the server gain to an individual transfer or kernel.

What changes if only the Poisson implementation changes? On a single CPU thread we held the workstation, three initializers, solver, checkpoints and eight batches fixed. One implementation applied the inverse through a type-I discrete sine transform. The other multiplied by the explicit sine basis along each axis and scaled by the same inverse eigenvalues. The two operators agreed to a maximum relative difference of 1.94e-15 on 16 probe vectors. All matched queries took identical iteration counts. Nevertheless, changing the inverse reduced the zero-start cost from 26.221 to 23.051 ms per query. The extra cost of physics-FNO over zero was 10.343 ms with the transform and 11.861 ms with the matrix implementation. The paired interval for the change in this extra cost included zero. This one-factor comparison isolates a software choice that the cross-machine comparison alone cannot isolate.

For readers implementing the method, the paper now defines the initial field, right-hand side, matrix and residual at their first use. The same stopping check follows the prediction all the way to the accepted answer. Readers can measure the work their residual-trained state saves and the cost of supplying it. Both quantities are needed to tell whether the next accepted solution will arrive sooner.

AuthorsOpenReview ↗

Which comparison should guide deployment?

For deployment, the runner-up matters. On the held-out server batches, physics-FNO plus Poisson-PCG was 12.872 ms per query faster than a fresh direct solve. Against a zero start with the same preconditioner, it saved just 0.201 ms. A service already running the latter solver has the smaller gain available. Your concern about the practical value of a sub-percent result led us to report the solver a user would replace.

A stream of new operators. In the selected workload each of the 32 queries in a batch has its own diffusion matrix, with 127 interior points per axis. The network weights are resident. There is no stored factorization to reuse when the next matrix arrives. We first screened 48 combinations of equation, resolution, batch size and learned model on exploratory groups. We then fixed the workload and physics-FNO checkpoint before timing eight fresh groups. Neither of the two discovery groups enters the confirmation. The design lets us estimate the gain on fresh tasks from the selected workload. It also explains why the 0.73% is specific to that solver choice and workload.

Seven paths faced the same inputs. Zero plus PCG took 27.469 ms per query, the coarse start plus PCG took 31.892 and the direct solver took 40.139. Physics-FNO took 27.267. Relative to the coarse initializer its paired saving was 4.624 ms per query, with a 95% interval from 4.544 to 4.702. The 95% interval against the direct path was 12.246 to 13.528 ms. Zero is cheap to supply and benefits from the same Poisson inverse, which makes it the harder comparison. The reported time is elapsed batch time divided by 32. It measures the cost per request when a batch is already available. It would give an unduly optimistic picture of latency for a service waiting to collect its first batch.

Which work enters the clock? All paths pay for assembly, right-hand-side preparation, initialization, any preconditioner setup, correction and the true-residual check. Neural paths pay for transfer and inference. Each path gets a warm-up, followed by three measured repetitions per group in randomized blocks. We take a median within each group and compare matched groups. All 5,376 measured query endpoints passed the same 1e-8 residual threshold. They come from 256 physical tasks across seven paths and three repetitions. The eight groups, rather than the number of endpoint receipts, supply the units for the paired timing interval. Model loading, source training, field generation and computing the independent reference lie outside this online measurement.

The absolute size of the advantage sets a clear requirement for amortization. At 0.201 ms saved per query, a single additional second spent before deployment would require about 5,000 queries of this kind to recover. This gives a preparation budget for a given query count. Training time itself remains unknown. In a separate within-process diffusion test at the same grid size, reloading the physics-FNO weights added 5.59 ms per query compared with keeping them resident. That test includes deserialization and device transfer in a running process. It should be interpreted in its own workload, but it shows why the lifetime of the loaded model belongs in a deployment report.

One matrix used repeatedly. When only the right-hand side changes, a direct solver can retain its factorization. We measured four matrix tasks at each grid with eight right-hand sides per matrix and charged setup once. At grid 127, diffusion took 6.46 ms per query for the cached direct path, 27.75 for zero plus PCG and 29.23 for physics-FNO. On advection-diffusion the corresponding times were 6.54, 122.56 and 121.14 ms. The latter comparison also illustrates a positive learned effect relative to zero that a deployment should still decline when a faster numerical path is available. Eight right-hand sides is the measured case in Table 15. Other query counts there project how the measured setup and per-query costs would accumulate.

One request at a time. We also timed serial queries. At grid 127, serial diffusion took 28.57 ms from zero and 29.89 ms with the physics-FNO start. A fresh direct solve took 40.87 ms. On serial advection-diffusion at that grid the fresh direct path took 41.14 ms, against 100.01 for the zero-start Poisson path and 102.22 for physics-FNO. Three details therefore matter for a simulation team. Does the matrix change at the next request? Can the model stay loaded? Can queries be batched within the required response time?

Section 6 and its appendix let a reader trace each number to that workload and acceptance rule. A team can repeat the comparison with its own batch size, operator reuse and reference implementation. The server result supplies a positive example with a measured margin. The fixed-matrix experiment demonstrates a larger opportunity in reusing classical numerical work. Both outcomes tell a practitioner what to preserve between requests and what to time before changing a running solver.

AuthorsOpenReview ↗

What the nonlinear experiment supports

The nonlinear experiment tests transfer with a different corrector. A network trained on one-dimensional cubic problems can supply an initial field that saves Newton updates on fresh tasks, including shifts in both parameters and forcing. A coarse numerical initializer is stronger in the same test. Both comparisons matter when choosing an initializer for a solver already in use.

What changes in this test? We solve a diffusion equation with a cubic reaction, homogeneous Dirichlet boundary conditions, positive diffusion and a nonnegative reaction coefficient. The operator is monotone and the discrete root is unique. We use damped Newton to correct the initial field, with banded linear solves for the Jacobian. The Jacobian now depends on the current iterate. A network output can begin closer to the root and still require several Newton updates before the residual meets the threshold. We apply the same acceptance test to zero, linearized, coarse-grid and learned starts. This setting lets us ask about work saved by a learned state at a well-defined target solution.

What transferred to new tasks? The six longer-trained FNOs use three training seeds under each of two objectives on the same source pool. After training we froze them and drew 56 fresh physical tasks, eight per stratum, with identifiers disjoint from the original 112 tasks. Table 31 reports grid 63 and a true relative residual tolerance of 1e-8. On the eight jointly shifted tasks the data-trained FNO took 5.875 Newton updates on average. Zero took 6.875. The paired saving of 1.000 step had a 95% task-bootstrap interval of 0.708 to 1.292. The residual-trained FNO took 5.667 steps, giving a saving of 1.208 against zero. The coarse start took 3.000. All 4,536 confirmation endpoints met the tolerance. The positive result against zero survived the new draw, and the classical comparison shows exactly how much more correction work could be avoided on these instances.

Which changes were easiest to handle? For fresh in-distribution tasks the data-trained FNO averaged 2.083 Newton updates, against 4.625 from zero and 2.000 from coarse. Its saving over zero was 2.542 steps, with an interval of 2.208 to 2.875. The same fitted states saved 1.292 steps in the low-diffusion stratum, 0.792 with stronger reaction and 0.750 under rough forcing. In the linear limit all three paths took one Newton update, despite different prediction errors. That limit serves as a useful check on the interpretation of a work metric. When a single accepted Newton step already solves the linear problem, better initial predictions have very little opportunity to change the count.

The extreme joint shift sets a boundary. There the data-trained model took 11.792 steps on average and zero took 11.125. The saving was -0.667, with a 95% interval from -1.668 to 0.418. The coarse start took 6.125. Eight fresh problems do not determine the sign of the learned-versus-zero difference with precision in that stratum. Each interval resamples physical tasks conditional on the fitted states. Counting three optimizer seeds, repeated timings or additional grids as independent tasks would exaggerate the support for a broader claim.

How much did the training recipe matter? We also retrained the six states on the original pool with a cosine schedule stretched from 1,200 to 6,000 updates. On the same original tasks the mean prediction error fell across all seven strata under both objectives. For the data-trained model the joint-shift saving over zero increased from 0.021 to 0.729 Newton steps. The separate 56-task confirmation then tested the longer-trained states without another training run. The two budgets use different schedules, so the comparison describes sensitivity to the full training recipe. Keeping that comparison separate from confirmation makes it possible to see what was observed on the original targets and what held on new ones.

What carries over from the linear study? The idea of measuring correction to a common tolerance transfers directly. For diffusion with CG, Figure 2 goes further and tests a mechanism by exchanging residual components in a fixed eigenspace between trained predictions. In damped Newton the Jacobian changes as the iterate moves. The cubic experiment measures a work difference. Establishing its mechanism needs a separate test. With multiple roots one would also need to check which solution the initializer reaches. The present family gives us a setting where the root is unambiguous, the numerical baseline is competitive and all starts can be judged at the same accepted solution.

Your concern about exposition also changed how we introduce this distinction. Section 2 defines what the solver receives from the model, what it computes on the target problem and how to recognize an accepted solution. Table 31 then gives a concrete answer for a fresh shifted problem. Prior learning can shorten the Newton correction, and a coarse solve can shorten it further.

AuthorsOpenReview ↗

Thank you for pressing on the practical claim

Thank you for asking us to change the Poisson implementation and for pointing out where the notation obscured the argument. Your questions led us to test the same 256 problems on a second machine and to separate the cost of producing a prediction from the work saved in correction. For a team with a working PDE solver, that distinction determines whether adding a model gets the next verified solution back sooner. If the revised evidence and clearer presentation address your concerns, would you consider raising your overall score? Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.