acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prediction: Extracting Preconditioning Structure from Fourier Neural Operators

15 views

Abstract

Neural operators provide rapid predictions, while classical iterative solvers deliver high-accuracy solutions, motivating growing interest in combining their strengths. Interpreting neural operators as learned approximations to inverse PDE operators naturally suggests Jacobian-based preconditioning. However, solution supervision alone does not ensure effective preconditioners, leading existing approaches to modify training objectives or impose additional constraints. Our theoretical analysis shows that Fourier neural operators (FNOs) trained for solution prediction can still contain recoverable preconditioning structure. Building on this insight, we introduce purification, a post-processing technique that extracts this structure from FNO Jacobians without retraining. Experiments on Poisson problems support our theoretical analysis. For Helmholtz problems, NOP reduces mean GMRES iterations by up to compared with unpreconditioned GMRES. For nonlinear convection–diffusion problems, NOP reduces mean cumulative GMRES iterations by up to compared with unpreconditioned Newton–GMRES. These results offer a new perspective on how neural operators can enhance iterative solvers through structure extraction.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Operator learning, Preconditioning, Krylov subspace methods, Spectral analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pgWhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes using Fourier neural operators trained with the standard loss to derive efficient preconditioners for solving a PDE. The paper theoretically analyzes the effectiveness of the method on neural operators with and without nonlinearities. The method shows empirically that the proposed post-processing technique yields significant improvements over using the Jacobian of the neural operator directly.

Critical strengths
  1. The theoretical results are strong. The results are quite robust to the nonlinearities that are often present in FNOs. Furthermore, the theoretical results also hold up in experiments.
  2. The experiments are very rigorous. The method has been tested on linear/non-linear PDEs, varying spatial dimensions, and varying boundary conditions.
Critical weaknesses
  1. The results of the paper would be most satisfying if purification outperformed the Fast Poisson Solver across all settings. Purification improves on the direct FNO Jacobian, which highlights the importance of the proposed method if we wanted to use the FNO Jacobian as a preconditioner. However, the motivation for using an FNO Jacobian at all seems lost in the experiments.
Questions for authors
  1. How did the Direct Jacobian perform in Darcy2D setup? The results have not been presented in Table 3.
  2. FPS seems to outperform the proposed method in most cases. What is the intuition behind this performance gap? Can this be supported theoretically?
3 comments
AuthorsOpenReview ↗

Response to Reviewer pgWh: Darcy Direct results, applicability, and limitations

Thank you for your thoughtful and constructive feedback. Below, we clarify the omission of the Darcy2D Direct results and discuss the applicability and significance of our approach.

1. Direct Jacobian results in Darcy2D (Q1)

This was an oversight in our presentation. Although the caption of Table 3 stated that Direct fails in all tests, we did not include its results in the table itself. We have revised Table 3 to include explicit Direct rows, showing 0/50 successes under both initializations in every Darcy2D setting.

2. Applicability and limitations: motivation and comparison with FPS (W1 & Q2)

We agree that FPS achieves lower iteration counts in most tested settings. The motivation for using the FNO Jacobian is that a model trained only for solution prediction may already contain useful information about the inverse PDE action beyond its prediction as an initial guess. FPS uses multipliers derived from a prescribed Poisson reference operator. Purification instead extracts problem-dependent multipliers from the learned source-to-solution map. For nonlinear FNOs, these multipliers can also depend on the input source. Comparing Direct with Purify shows that structural extraction can make this learned information useful for preconditioning without retraining.

FPS is particularly effective in diffusion-dominated problems because its multipliers capture the dominant diffusion structure. Purification's flexibility does not ensure that every mode is accurately learned. Modes with little energy in the training solutions contribute little to the prediction loss, so solution supervision may constrain their relative multiplier errors weakly. This is reflected in the relative mode-wise errors in Figure 3 and quantified by the frequency-dependent error bound in Corollary 2.3. Thus, the gap with FPS can arise even when purification recovers useful preconditioning information.

In the revisions, we added a Helmholtz2D comparison with the Graph Neural Preconditioner (GNP), a learned preconditioner that constructs a nonlinear approximate inverse action from the sparse system matrix. Table 1 in Section 3.3 reports the results:

Table 1. Helmholtz2D: success count / mean LI over successful solves (50 sources).

k0=16k_0=16k0=20k_0=20k0=24k_0=24
Precond.Zero Init.Neural Init.Zero Init.Neural Init.Zero Init.Neural Init.
None0 / −-7 / 1678.710 / −-0 / −-0 / −-0 / −-
FPS50 / 54.9450 / 54.8650 / 76.8450 / 75.5450 / 101.8450 / 99.78
Direct0 / −-0 / −-0 / −-0 / −-48 / 570.6248 / 549.10
Purify50 / 135.1250 / 135.0250 / 169.3250 / 167.7450 / 223.4450 / 220.84
GNP50 / 231.9450 / 232.3250 / 278.7850 / 276.9850 / 344.0650 / 339.22

In this setting, Purify performs better than GNP in terms of iteration count, although FPS achieves the lowest iteration counts.

Strong nonlinearity or transport can make purification less effective when it discards important mode coupling or phase information. As discussed in Section 4, our work offers insight for more complex PDEs by showing that useful preconditioning structure can be recovered from neural operators trained only for solution prediction. This insight motivates the development of extraction methods that retain useful mode coupling while preserving a structure suitable for efficient preconditioning. Beyond FNOs, other prediction-trained neural operators may also encode useful preconditioning information. Understanding their learning biases could guide its extraction instead of imposing additional training constraints.

Thank you for your response. The inclusion of the missing Darcy2D results addresses my first question. I also appreciate the explanation of the performance gap between FPS and purification, particularly the discussion of frequency-dependent errors.

While I still have some reservations about the practical advantages of purification over classical preconditioners, I recognize that the primary contribution of this work is demonstrating that useful preconditioning structure can be extracted from neural operators trained solely for solution prediction.

Overall, I believe the authors have adequately addressed my questions, and I will maintain my original score.

Thank you for your thoughtful comments and constructive feedback. We sincerely appreciate your recognition of the central contribution of our work and are glad that our clarifications have addressed your questions.

Reviewer_vNgKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether FNO trained only to predict PDE solutions can also be used to speed up iterative solvers. The authors find that the network’s Jacobian cannot be used directly as a preconditioner. They propose a post-processing step called purification, which extracts the Fourier-diagonal part of this Jacobian and turns it into a cheap FFT-based preconditioner, with no retraining. Combined with using the network’s prediction as the initial guess, this gives a framework for accelerating GMRES and Newton–GMRES solvers with prediction-trained neural operators.

Critical strengths
  1. The paper asks an interesting question: can a neural operator trained only to predict solutions still help with preconditioning? Here the authors reuse an already trained model without retraining, which is cheap and practical.
  2. Theorems 2.2 and 2.4 give a clear theoretical link between prediction accuracy and preconditioning in the simplified linear case, and they explain why purification makes sense.
Critical weaknesses
  1. The theory does not cover the experiments. Theorems 2.2 and 2.4 assume a linear FNO without activations and a Fourier-diagonal operator. Only Poisson1D meets these assumptions. All 2D models use GELU, zero padding and spatial RMS normalization, and the 2D problems have Dirichlet boundaries (Helmholtz, Darcy), are indefinite (Helmholtz) or are nonlinear (Darcy, Convec2D). So the theory says nothing about the cases where purification is actually needed.
  2. All 2D experiments use 64×6464 \times 64 grids, and the theory is checked on a single 1D grid (n=128n = 128). It is unclear how the method behaves on larger grids (2048×20482048 \times 2048, 4096×40964096 \times 4096), where iterative solvers actually matter. In Table 10 the condition number κ2\kappa_2 after purification still grows with resolution (7.51→29.28→116.297.51 \rightarrow 29.28 \rightarrow 116.29).
  3. The FPS (fast Poisson solver) needs fewer iterations on Helmholtz (20.0020.00 vs 49.4049.40 at k0=8k_0 = 8, 56.5456.54 vs 293.75293.75 at k0=16k_0 = 16) and on Darcy (1100.501100.50 vs 1675.421675.42 at σg=40\sigma_g = 40). On Convec2D it is faster in total time (0.2100.210 s vs 0.2590.259 s, Table 11). There is no comparison with the learned preconditioners from the Related Work section.
  4. The Helmholtz test is too easy. Only small wavenumbers are used (k0∈8,12,16k_0 \in {8, 12, 16}), and FPS, which is just a Laplacian preconditioner, converges in 19–5719–57 iterations. This regime does not show that the method can handle hard indefinite problems.
Questions for authors

See the weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer vNgK (1/3): Theoretical scope and extension

Thank you for your careful review and helpful suggestions. Our response is organized as follows: theoretical scope (Comment 1), resolution transfer and extraction cost (Comment 2), and baseline comparisons and harder Helmholtz tests (Comment 3).

1. Theoretical scope and extension to more general PDEs (W1)

We agree that the theoretical guarantees do not fully cover the experimental settings. The analysis under restrictive assumptions nevertheless provides the key insight behind purification: Theorem 2.2 links prediction accuracy to useful preconditioning structure in the Fourier multipliers, motivating its extraction from more general FNO Jacobians. The experiments further demonstrate the applicability of purification in general settings.

Extending this analysis to FNOs with nonlinear activations and padding is more challenging. Activation derivatives depend on the input and vary across spatial locations, while padding can also disrupt Fourier diagonalization. The Jacobian is therefore generally no longer Fourier-diagonal, so the argument in Theorem 2.2 does not extend directly. Relating prediction error to preconditioning quality in this nonlinear setting requires new analytical tools.

To broaden the theoretical scope, we have added Theorem 2.5 in Section 2.3, which relaxes the assumptions on the PDE matrix:

Theorem 2.5. Let Φ\Phi be the unitary spatial DFT and J∈Rn×nJ\in\mathbb R^{n\times n}. Define

dξ=(ΦJΦ∗)ξξ≠0,M=Φ∗diag⁡(∣dξ∣)Φ.\displaystyle d_\xi=(\Phi J\Phi^*)_{\xi\xi}\ne0, \qquad M=\Phi^*\operatorname{diag}(|d_\xi|)\Phi.

Write Ah=Aref+EhA_h=A_{\mathrm{ref}}+E_h, where Aref=Φ∗diag⁡(aref,ξ)ΦA_{\mathrm{ref}}=\Phi^*\operatorname{diag}(a_{\mathrm{ref},\xi})\Phi is real SPD with aref,ξ>0a_{\mathrm{ref},\xi}>0, and EhE_h is any real matrix. Let

g‾ref=min⁡ξaref,ξ∣dξ∣,g‾ref=max⁡ξaref,ξ∣dξ∣,\displaystyle \underline g_{\mathrm{ref}}=\min_\xi a_{\mathrm{ref},\xi}|d_\xi|, \qquad \overline g_{\mathrm{ref}}=\max_\xi a_{\mathrm{ref},\xi}|d_\xi|,

cref=2g‾ref+g‾ref,qref=g‾ref−g‾refg‾ref+g‾ref,\displaystyle c_{\mathrm{ref}}=\frac{2}{\overline g_{\mathrm{ref}}+\underline g_{\mathrm{ref}}}, \qquad q_{\mathrm{ref}}=\frac{\overline g_{\mathrm{ref}}-\underline g_{\mathrm{ref}}}{\overline g_{\mathrm{ref}}+\underline g_{\mathrm{ref}}},

q~=qref+cref∣MEh∣2.\displaystyle \widetilde q=q_{\mathrm{ref}}+c_{\mathrm{ref}}\\|ME_h\\|_2.

Then the worst-case one-step GMRES residual factor satisfies

Γ(MAh)≤min⁡1,q~.\displaystyle \Gamma(MA_h)\leq\min\\{1,\widetilde q\\}.

If q~<1\widetilde q<1, left-preconditioned GMRES satisfies

∣r~k∣2≤q~k∣r~0∣2,r~k=Mrk,\displaystyle \\|\widetilde r_k\\|_2\leq\widetilde q^{k}\\|\widetilde r_0\\|_2, \qquad \widetilde r_k=Mr_k,

where rkr_k is the true residual after kk total Krylov steps, including restarts.

The actual PDE matrix may therefore be nonsymmetric and couple Fourier modes, with convergence guaranteed when its deviation from the reference is sufficiently small after preconditioning. The proof is given in Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer vNgK (2/3): Resolution transfer and extraction cost

Regarding W2, we report resolution-transfer experiments and analyze the extraction cost.

2. Resolution dependence and scalability (W2)

Resolution transfer for linear Poisson1D. Appendix C.4, Table 10 evaluates one randomly selected test source (sample 817) at n=128,256,512n=128,256,512 with the same FNO trained at n=128n=128. Purified condition numbers increase from 7.51 to 29.28 to 116.29, while unpreconditioned values are 1660.38, 6640.52, and 26561.07: reductions of approximately 221×221\times, 227×227\times, and 228×228\times. Thus, purification inherits FNO's resolution-transfer capability and preserves the relative conditioning benefit for this linear test case.

Resolution transfer for nonlinear Convec2D. For nonlinear problems, resolution-transfer performance can deteriorate with refinement. In the revisions, we added an experiment in Appendix C.6 (Table 12). For Convec2D with Re=500Re=500, we fix the FNO trained on a 64×6464\times64 grid and evaluate it on 64×6464\times64, 128×128128\times128, and 256×256256\times256 grids without retraining, interpolating the same 10 source cases to finer grids. Results are summarized below.

Table 12. Convec2D at Re=500Re=500.

GridMethodZero init.: SuccessNILINeural init.: SuccessNILI
64×6464\times64None0/10−-−-6/106.672076.00
64×6464\times64FPS0/10−-−-9/104.67363.33
64×6464\times64Purify0/10−-−-9/104.78228.22
128×128128\times128None1/1011.005776.004/1011.753305.50
128×128128\times128FPS1/109.00396.0010/104.90348.90
128×128128\times128Purify3/1013.00658.679/105.22406.00
256×256256\times256None0/10−-−-0/10−-−-
256×256256\times256FPS3/1014.67714.3310/105.20366.80
256×256256\times256Purify7/1013.861488.4310/105.20944.60

Purification provides effective preconditioning across all three grids with neural initialization. However, mean cumulative LI increases from 228.22 to 944.60, whereas FPS remains near 350--370. This is consistent with coarse-to-fine transfer: coarse-grid training provides no direct supervision for newly resolved Fourier directions, which also couple to coarser modes in nonlinear convection, introducing interactions not explicitly represented on the training grid.

Extraction cost and scalability. Table 13 reports timings (s):

GridPurify assemblyIterationTotalFPS total
64×6464\times640.16410.1690.3360.265
128×128128\times1282.12350.2852.4100.604
256×256256\times25656.11170.49556.6092.084

Table 13 shows that Purify's assembly time grows rapidly with resolution: from 0.1641 s at 64×6464\times64 to 2.1235 s at 128×128128\times128 and 56.1117 s at 256×256256\times256. In comparison, its iteration time increases from 0.169 s to 0.495 s. Assembly therefore dominates the total time on finer grids and is the main computational bottleneck for scalability. This growth is consistent with the complexity analysis added in Section 4:

On a tensor grid with naxisn_{\mathrm{axis}} points along each of dd spatial dimensions (n=naxisdn=n_{\mathrm{axis}}^d), a multidimensional FFT costs O(dnaxisdlog⁡naxis)O(d n_{\mathrm{axis}}^d\log n_{\mathrm{axis}}). For fixed FNO depth and channel width, each automatic-differentiation JVP has this scaling, and extracting all Fourier-diagonal entries requires O(naxisd)O(n_{\mathrm{axis}}^d) such evaluations, giving total work O(dnaxis2dlog⁡naxis)O(d n_{\mathrm{axis}}^{2d}\log n_{\mathrm{axis}}). This cost grows rapidly with both resolution and spatial dimension.

On our current hardware, the extraction cost will outweigh the iteration savings for experiments on 2048×20482048\times2048 or 4096×40964096\times4096 grids. However, the directional evaluations are independent and can be batched or executed in parallel. GPUs with higher compute throughput and larger memory could therefore substantially reduce extraction time, although the attainable speedup remains to be quantified. Although faster extraction is needed for practical scaling, these experiments demonstrate purification's resolution-transfer capability.

AuthorsOpenReview ↗

Response to Reviewer vNgK (3/3): Applicability and harder Helmholtz tests

In response to W3 and W4, we clarify purification's applicability and limitations, report harder Helmholtz tests, and add a comparison with the Graph Neural Preconditioner (GNP).

3. Applicability and limitations (W3)

Purification retains a Fourier-diagonal approximation to the learned Jacobian. We expect purification to perform well on diffusion-dominated problems where a convolution operator captures the dominant inverse action. Strong nonlinearity or transport can make purification less effective when the discarded mode coupling or phase information is important. As discussed in Section 4, our work offers insight for more complex PDEs by showing that useful preconditioning structure can be recovered from neural operators trained only for solution prediction. This insight motivates the development of extraction methods that retain useful mode coupling while preserving a structure suitable for efficient preconditioning. Beyond FNOs, other prediction-trained neural operators may also encode useful preconditioning information. Understanding their learning biases could guide its extraction instead of imposing additional training constraints.

4. Harder Helmholtz benchmark and comparisons with GNP (W4)

In the revisions, we increased the base wavenumbers from k0∈{8,12,16}k_0\in\{8,12,16\} to k0∈{16,20,24}k_0\in\{16,20,24\} to test more strongly indefinite Helmholtz systems. We also added a comparison with GNP, a learned preconditioner that constructs a nonlinear approximate inverse action from the sparse system matrix. All itertive methods use FGMRES, as required by GNP's nonlinear preconditioning action. For a fair comparison, GNP uses the same FNO prediction for neural initialization, since GNP itself provides a preconditioning action rather than a solution prediction. Table 1 in Section 3.3 reports the results:

Table 1. Helmholtz2D: success count / mean LI over successful solves (50 sources).

k0=16k_0=16k0=20k_0=20k0=24k_0=24
Precond.Zero Init.Neural Init.Zero Init.Neural Init.Zero Init.Neural Init.
None0 / −-7 / 1678.710 / −-0 / −-0 / −-0 / −-
FPS50 / 54.9450 / 54.8650 / 76.8450 / 75.5450 / 101.8450 / 99.78
Direct0 / −-0 / −-0 / −-0 / −-48 / 570.6248 / 549.10
Purify50 / 135.1250 / 135.0250 / 169.3250 / 167.7450 / 223.4450 / 220.84
GNP50 / 231.9450 / 232.3250 / 278.7850 / 276.9850 / 344.0650 / 339.22

Purify and GNP both solve all 50 sources at every tested wavenumber under both initializations. Although FPS achieves the lowest iteration counts, these results suggest that purification is more suitable than GNP for this Helmholtz setting.

Reviewer_ayWdOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how to combine the strengths of neural operators and classical iterative solvers. Motivated by some theory, the paper introduces a post-processing technique that extracts some preconditioning structure from FNO Jacobians without retraining. Several experiments are implemented to support theoretical analysis and shows the benefit of preconditioning. The authors hoped to provide some new perspectives on how neural operators can speed up iterative solvers by extracting some special structure.

Critical strengths
  1. Combining modern machine learning approaches, such as Fourier neural operator, with traditional numerical solvers, to gain the benefits of both (efficiency and accuracy) has been a really important and fundamental problem, and will provide insights on combining between other modern machine learning techniques and traditional numerical solvers in the future.
  2. The idea behind this paper looks very interesting to me: in addition to using the predictions of Fourier neural operator as warm starts for traditional numerical solvers, this paper uses an interesting preconditioning as another important information from Fourier neural operator to help with the learning, so that this is a deeper connection between the two. Also, the paper's theoretical analysis is supported by numerical experiments.
Critical weaknesses
  1. The way of removing off-diagonal components does not look well-motivated enough to me, in particular for more general PDEs. The theory only shows under restrictive cases, the prediction accuracy can be guaranteed conditioned on good preconditioning quality. Also, in general, functions being close to each other does not necessarily mean their derivatives are close to each other.
  2. I think a more systematic quantification of the benefits we can gain from spending the extra effort in computing this preconditioning is needed. Also, some insights on how the magnitude of benefit depends on the key parameters of the PDEs will be helpful.
Questions for authors
  1. Can you discuss more on for what type of PDEs (e.g., depending on the source and strength of nonlinearity), you expect your method of preconditioning works better than other approaches, and when not? Even some intuitive descriptions would help.
  2. Can you relax some assumptions in the theory, so that it better matches with the PDE settings in the numerical results?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ayWd (1/3): Motivation and applicability of purification

Thank you for your thoughtful and constructive feedback. Our response is organized as follows: motivation and applicability (Comment 1), the theoretical extension (Comment 2), and computational cost and PDE parameter dependence (Comment 3).

1. Motivation, applicability, and limitations of purification (W1, Q1)

We agree that small prediction error does not imply small derivative error. Our starting point is that neural operators learn source-to-solution maps, which naturally suggests using their input Jacobians as approximations to inverse PDE Jacobians for preconditioning. However, direct Jacobian preconditioning fails in most tested settings, highlighting that prediction accuracy alone does not guarantee an effective preconditioning action. This gap motivates us to analyze the neural operators' architecture. For simplicity, we first study an FNO without activations or padding and identify its Fourier-diagonal Jacobian structure. For a PDE matrix that is also diagonal in the Fourier basis, simultaneous diagonalization and Parseval's identity allow us to relate prediction error to the eigenvalue distribution of the preconditioned system. These simplified analyses motivated our purification approach.

Motivation for removing off-diagonal components (W1). These simplified analyses inspire us to project the input Jacobian of a general FNO onto the space of Fourier-diagonal matrices. Deleting the off-diagonal entries in the Fourier basis is the orthogonal projection with respect to the Frobenius inner product. Experiments show that nonlinear activations and padding can disrupt the Jacobian's Fourier-diagonal structure, while purification recovers useful preconditioning information.

From the perspective of preconditioner structure, Fourier-diagonal operators form a widely used class of effective preconditioners. Moreover, this structure is computationally efficient for preconditioning. Once the multipliers are extracted, each application requires only a forward FFT, scalar multiplication at each frequency, and an inverse FFT. This gives a matrix-free preconditioning action, without assembling or storing a dense Jacobian.

Applicability and limitations (Q1). However, removing off-diagonal components also discards mode coupling. Strong nonlinearity or transport can make purification less effective when it discards important mode coupling or phase information. We expect purification to perform well on diffusion-dominated problems where a convolution operator captures the dominant inverse action. In the revision, we have added a comparison with the Graph Neural Preconditioner (GNP), a learning-based preconditioner, on Helmholtz2D in Section 3.3. The results are reproduced below.

Table 1. Helmholtz2D: success count / mean LI over successful solves (50 sources).

k0=16k_0=16k0=20k_0=20k0=24k_0=24
Precond.Zero Init.Neural Init.Zero Init.Neural Init.Zero Init.Neural Init.
None0 / −-7 / 1678.710 / −-0 / −-0 / −-0 / −-
FPS50 / 54.9450 / 54.8650 / 76.8450 / 75.5450 / 101.8450 / 99.78
Direct0 / −-0 / −-0 / −-0 / −-48 / 570.6248 / 549.10
Purify50 / 135.1250 / 135.0250 / 169.3250 / 167.7450 / 223.4450 / 220.84
GNP50 / 231.9450 / 232.3250 / 278.7850 / 276.9850 / 344.0650 / 339.22

Purify requires fewer FGMRES iterations than GNP, while FPS achieves the lowest iteration counts in this Helmholtz comparison.

FNO assumes translation invariance of the learned kernel, yielding a convolution structure that can be implemented in Fourier space. This prior may be less suitable for PDEs with strong mode coupling. As discussed in Section 4, our work offers insight for more complex PDEs by showing that useful preconditioning structure can be recovered from neural operators trained only for solution prediction. This insight motivates the development of extraction methods that retain useful mode coupling while preserving a structure suitable for efficient preconditioning. Beyond FNOs, other prediction-trained neural operators may also encode useful preconditioning information. Understanding their learning biases could guide its extraction instead of imposing additional training constraints.

AuthorsOpenReview ↗

Response to Reviewer ayWd (2/3): Extension of the theoretical analysis

To answer Q2, we present an extension of the theoretical analysis to more general PDEs.

2. Extension to more general PDEs (Q2)

For more general PDEs, we have added Theorem 2.5 in Section 2.3 to extend the analysis beyond SPD, Fourier-diagonal PDE matrices. We reproduce the statement here for convenience, making the definitions explicit.

Theorem 2.5. Let Φ\Phi be the unitary spatial DFT and J∈Rn×nJ\in\mathbb R^{n\times n}. Define

dξ=(ΦJΦ∗)ξξ≠0,M=Φ∗diag⁡(∣dξ∣)Φ.\displaystyle d_\xi=(\Phi J\Phi^*)_{\xi\xi}\ne0, \qquad M=\Phi^*\operatorname{diag}(|d_\xi|)\Phi.

Write Ah=Aref+EhA_h=A_{\mathrm{ref}}+E_h, where Aref=Φ∗diag⁡(aref,ξ)ΦA_{\mathrm{ref}}=\Phi^*\operatorname{diag}(a_{\mathrm{ref},\xi})\Phi is real SPD with aref,ξ>0a_{\mathrm{ref},\xi}>0, and EhE_h is any real matrix. Let

g‾ref=min⁡ξaref,ξ∣dξ∣,g‾ref=max⁡ξaref,ξ∣dξ∣,\displaystyle \underline g_{\mathrm{ref}}=\min_\xi a_{\mathrm{ref},\xi}|d_\xi|, \qquad \overline g_{\mathrm{ref}}=\max_\xi a_{\mathrm{ref},\xi}|d_\xi|,

cref=2g‾ref+g‾ref,qref=g‾ref−g‾refg‾ref+g‾ref,\displaystyle c_{\mathrm{ref}}=\frac{2}{\overline g_{\mathrm{ref}}+\underline g_{\mathrm{ref}}}, \qquad q_{\mathrm{ref}}=\frac{\overline g_{\mathrm{ref}}-\underline g_{\mathrm{ref}}}{\overline g_{\mathrm{ref}}+\underline g_{\mathrm{ref}}},

q~=qref+cref∣MEh∣2.\displaystyle \widetilde q=q_{\mathrm{ref}}+c_{\mathrm{ref}}\\|ME_h\\|_2.

Then the worst-case one-step GMRES residual factor satisfies

Γ(MAh)≤min⁡1,q~.\displaystyle \Gamma(MA_h)\leq\min\\{1,\widetilde q\\}.

If q~<1\widetilde q<1, left-preconditioned GMRES satisfies

∣r~k∣2≤q~k∣r~0∣2,r~k=Mrk,\displaystyle \\|\widetilde r_k\\|_2\leq\widetilde q^{k}\\|\widetilde r_0\\|_2, \qquad \widetilde r_k=Mr_k,

where rkr_k is the true residual after kk total Krylov steps, including restarts.

The actual PDE matrix may therefore be nonsymmetric and couple Fourier modes, with convergence guaranteed when its deviation from the reference is sufficiently small after preconditioning. The proof is given in Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer ayWd (3/3): Computational payoff and dependence on PDE parameters

Regarding W2, we quantify the computational benefits and extraction cost, and discuss how the benefits depend on PDE parameters.

3. Quantifying the computational payoff and its dependence on PDE parameters (W2)

Computational benefit and extraction cost. At Re=250Re=250 and 500500, Purify achieves lower mean iteration time than FPS with the same FNO prediction as initialization. Table 11 (Appendix C.5) reports the timing breakdown:

ReReMethodSuccessIteration (s)Assembly / extraction (s)Total (s)
250FPS49/500.0960.0440.141
250Purify50/500.0830.1560.239
500FPS47/500.1660.0440.210
500Purify47/500.1020.1560.259

The iteration-time savings are currently offset by the higher extraction cost, so Purify does not yet achieve lower total time than FPS in these tests.

In the revised manuscript, we added a resolution-transfer study in Appendix C.6, which shows how this trade-off changes with resolution. We fix the FNO trained at Re=500Re=500 on a 64×6464\times64 grid and evaluate the same 10 source cases, interpolated to finer grids, without retraining. Purify's extraction time rises from 0.1641 s to 2.1235 s and then 56.1117 s on 64×6464\times64, 128×128128\times128, and 256×256256\times256 grids, respectively (Table 13). The rapid growth is consistent with the complexity analysis in Section 4, also discussed in Appendix C.6:

On a tensor grid with naxisn_{\mathrm{axis}} points along each of dd dimensions (n=naxisdn=n_{\mathrm{axis}}^d), each automatic-differentiation Jacobian-vector product costs O(dnaxisdlog⁡naxis)O(d n_{\mathrm{axis}}^d\log n_{\mathrm{axis}}) for fixed FNO depth and width. Exact extraction requires O(naxisd)O(n_{\mathrm{axis}}^d) such evaluations, giving total work O(dnaxis2dlog⁡naxis)O(d n_{\mathrm{axis}}^{2d}\log n_{\mathrm{axis}}): quadratic in the number of unknowns, up to an FFT logarithmic factor.

However, these directional evaluations are independent and can be batched or executed in parallel. GPUs with higher compute throughput and larger memory could therefore substantially reduce extraction time, although the attainable speedup remains to be quantified. Although further reductions in extraction cost are needed for practical scaling, these experiments demonstrate that Purify can reduce iteration time.

Dependence on PDE parameters. With neural initialization, the sweeps show two distinct trends:

  • Coefficient heterogeneity (Darcy2D, Table 3): as the log-permeability scale increases through σg=10,20,40\sigma_g=10,20,40, Purify's reduction in mean cumulative linear iterations relative to no preconditioning decreases from 5.39×5.39\times to 4.13×4.13\times to 2.73×2.73\times, with all 50 sources solved. FPS requires fewer iterations throughout. This is consistent with discarded mode coupling becoming more consequential as heterogeneity increases.
  • Nonlinear transport (Convec2D, Table 4): With neural initialization, Purify achieves lower mean cumulative linear iteration counts among successful solves than FPS at Re=250Re=250 and 500500 (263.23 versus 435.77 at Re=500Re=500). FPS uses a fixed Poisson inverse, whereas Purify extracts problem- and source-dependent Fourier multipliers from the prediction-trained model. The results show that this extracted information can yield lower reported iteration counts than the fixed Poisson reference. As nonlinearity increases further, a Fourier-diagonal approximation may no longer capture the dominant inverse action, motivating extraction methods that preserve useful mode coupling or the use of other neural operators.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of manuscript revisions: Extended theory, a learned preconditioner baseline, and resolution-transfer experiments

We thank all reviewers for their thoughtful feedback. We revised the manuscript to address their concerns. Major additions and revisions are marked in \color[RGB]{0,70,160}{\text{blue}}. Below, we summarize the changes and their locations for convenient reference.

1. Extended theory beyond exactly Fourier-diagonal PDE matrices (Section 2.3, Theorem 2.5; proof in Appendix A.5).

We added a perturbation analysis around an SPD Fourier-diagonal reference operator. The PDE matrix is no longer required to be symmetric or diagonal in the Fourier basis. The theorem bounds the GMRES residual factor in terms of the operator deviation after preconditioning. It guarantees geometric contraction of the left-preconditioned residual when this deviation is sufficiently small. This extends the original analysis.

2. Harder Helmholtz experiments and a learned baseline (Section 3.3, Tables 1–2; training results in Appendix C.3.2).

We replaced the previous wavenumber sweep of k0∈{8,12,16}k_0\in\{8,12,16\} with k0∈{16,20,24}k_0\in\{16,20,24\} and added a comparison with the Graph Neural Preconditioner (GNP). Purify requires fewer iterations on average than GNP, while FPS achieves the lowest iteration counts. We also updated the Fourier-diagonal energy and one-step residual diagnostics for these settings.

3. New resolution-transfer experiments without retraining (Appendix C.6, Table 12).

For the Convec2D equation with Re=500Re=500, we fix the FNO trained on a 64×6464\times64 grid and evaluate the same 10 source cases, interpolated to 128×128128\times128 and 256×256256\times256. With neural initialization, Purify succeeds on 9/10, 9/10, and 10/10 cases across the three grids. Its mean cumulative linear iterations increase from 228.22 to 944.60, and FPS is more efficient on the finer grids. These results demonstrate useful resolution transfer for the tested interpolated sources.

4. Computational complexity and scalability (Appendix C.6, Table 13; Section 4).

We added a computational complexity analysis of preconditioner extraction and application, together with timing results on finer grids. The analysis and timings identify exact extraction as the main scalability bottleneck.

Our central contribution is that FNOs trained only for solution prediction can contain useful preconditioning information beyond a warm start, even when their full Jacobians are ineffective. The revisions strengthen this evidence and clarify the scope and computational limits of the current extraction method. More broadly, this insight motivates extraction methods that retain useful mode coupling for more complex PDEs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.