acceptodds
Under review as a conference paper at ICLR 2027

From Physics-Informed Losses to Equation-Derived Priors and Representations for Neural Operators

Abstract

When governing equations are known but solution data are scarce, equation knowledge can be used not only as a training constraint but also as an inductive bias for the neural operator. We study three ways of exposing equation information: enforcing the equation as a loss, using it to construct an equation-derived prior, and injecting the structure it reveals into the residual. Our idea is to use the governing equation to reduce the learning problem before training. Across the benchmarks that admit a physics-informed loss, the equation-derived prior compares favorably with imposing the same equation as a training loss, and some problems benefit further from structural injection beyond the prior. These results suggest that equation knowledge is more effective when it is used to shape the representation before training than when used solely as a training constraint.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neural Operators, Scientific Machine Learning, Operator Learning, PDEs, Inductive Bias, Physics-Informed Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VYxhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors present a method to use equation-derived priors to structurally embed approximate or exact ground truth equation priors into neural operator architectures to improve performance. The authors showcase their results of learning a physics residual using FNO and DeepONet and their results on learning with structurally-embedded equation priors using DeepONet. The results demonstrate improvements over baselines.

Critical strengths

The experiments are thorough and applied to a wide range of systems and benchmarks. I appreciate error comparisons against training budgets (Figure 2) and the diverse set of problems (e.g., Figure 3) that the authors compared against.

Critical weaknesses

The main weakness of this paper is its scope. Please correct me if I misunderstood, but the non-DeepONet experiments appear to be only learning a residual against a given prior using an FNO, which I would argue is a good baseline but perhaps not particularly novel as a proposed method. On the other hand, injecting structural priors in the architecture is a novel and interesting area, but the experiments appear to me to be primarily useful only for DeepONet architectures. Again, please correct me if I misunderstood here.

If my understanding is correct, I would concretely recommend that the authors present more general recipes for incorporating equation priors into neural operator architectures and perhaps explore different types of architectures, such as attention-based architectures or ones using different bases (non-Fourier). This would strengthen the paper substantially.

Also, physics-informed or “physics-embedded” methods are largely useful up to their concrete applications. If computational resources and time permit, I would recommend that the authors expand their experimental set beyond simple toy settings and explore larger real-world problems (e.g., weather forecasting, where there is a substantial amount of dataset and training frameworks, comes to mind, although I understand that this may be outside the scope of a methods paper).

Furthermore, in my personal opinion, the merits of this paper are overshadowed by lack of clarity in the writing, and much of the text reads as though it may have been AI-generated. While I cannot verify that LLMs generated the text, I do believe that the style seems close to that of frontier language models. Below I have some selected examples of what I am referring to:

Instances of terminology that should be changed or clarified:

  • “Learner” in paragraph 4
  • “Instance-side factor” (line 140)

A few sentences where AI-like phrasing may obscure understanding or sound unprofessional:

  • “For a direct prior this residual is not an arbitrary leftover.” (line 150)
  • “leaving all four at their data-driven defaults gives a raw operator” — what is a “raw operator”?
  • “Two qualifications.” - line 357
  • Figure 3 caption: “What the rank-two prior removes.”
  • “The gains are conditional.” (Line 463)

I understand that some of these may be stylistic choices, and I do not intend to penalize the authors for these choices, although I do personally recommend that they change some wordings to maintain an academic tone. My main concern is when stylistic writing choices begin to conflict with reader understanding, which is particularly important in Sections 2 and 3.

Minor notes:

  1. Introduction: “and even where it is exact, a lower residual loss does not guarantee a more accurate solution.” Can you expand on this in the text? A reader may ask how the result could be inaccurate if the ground truth equation is satisfied.
  2. It would be useful to include some examples of where “direct” and “indirect” priors could be derived from in section 2.
Questions for authors
  1. Does the structural injection only apply to DeepONet? It seems like it relies on the trunk/branch structure and is not architecture-agnostic.
  2. What are the specific instance/spatial factors and structural injection basis functions that you use for your experiments? It is possible I missed them in the main text, but I believe these should be mentioned clearly in Section 3, at least for 1-2 prototypical systems.
8 comments
AuthorsOpenReview ↗

Summary of our response to Reviewer VYxh

We thank the reviewer for the careful reading. The reviewer's reading was accurate for the submitted version; we address the main limitation with new experiments.

ConcernResponsePart
Main weakness / Q1: injection is DeepONet-onlyImplemented in GNOT and Transolver (general comment, R1–R4). Beats same-width random and POD bases on 4/6 benchmarks in GNOT, 2/6 in Transolver. Residual learning is a comparison point, not our method1/2
Recommendation: general recipe, non-Fourier basesRecipe in the general comment; no backbone or basis is Fourier1–2/2
Recommendation: larger real-world problemsAgreed; will be named as future work1–2/2
Q2: which factors and bases?Explicit forms for Pipe and annulus; will be moved into Sec. 31/2
Writing and terminologyRevision table for every flagged phrase; Secs. 2–3 will be rewritten2/2
Minor notes 1–2Both will be expanded with examples2/2
AuthorsOpenReview ↗

Response to Reviewer VYxh (1/2)

Main weakness

Reviewer’s comment: Main weakness. The non-DeepONet experiments only learn a residual with an FNO, which is a good baseline but not novel, and structural injection appears useful only for DeepONet.

Response.

The reviewer reads both points correctly for the submitted version, and we address the second with new experiments.

Residual learning with FNO. We agree that learning a residual on top of a prior is not new as a method, and we do not propose it as one. In the paper it is Level 2, one of three ways of exposing the same equation that we compare: as a loss, as a prior, and as a basis. The contribution is the comparison and what it shows. Used as a loss, the equation helps little, and its bias toward the approximate equation does not shrink with more data (our response to Reviewer ycrY, W2–Q2, shows this pull directly). Used as a prior, it helps in proportion to how much of the solution it already resolves, and a strong backbone such as FNO can close the gap. Used as a basis, it gives the largest and most robust gains, and where the basis carries structure the backbone does not have, these survive a change of backbone. The FNO experiments are therefore there to test the first two uses under a strong backbone, not to propose residual learning.

Structural injection beyond DeepONet. This was a fair limitation, and we address it with new experiments that we will add to the paper: the general comment on attention-based backbones applies the injection to GNOT and Transolver on all six benchmarks with an equation-derived basis (Tables R1–R2 and R3).

Presentation. We take the low presentation score seriously. In the revised PDF, the three uses of F0F_0 are placed side by side in Fig. 1 and Table 1 with one notation throughout, a new table in Sec. 3 lists the factors of every benchmark together with the Pipe and annulus examples, and the per-benchmark constructions stay in the appendix, so that the main text follows a single line: where the equation enters, and what each choice buys.

Q1, Q2

Reviewer’s comment: Q1. Does structural injection apply only to DeepONet, given its reliance on the trunk/branch structure? Q2. Which instance and spatial factors and basis functions are used, at least for one or two prototypical systems?

Response to Q1.

No; see the general comment on attention-based backbones and the response to the main weakness above.

Response to Q2.

We agree that these belong in Sec. 3 and will move them there. Each basis column factorizes as Φ(l,j)(x;a)=Flj(af) Sl(x;ag)\Phi_{(l,j)}(x;a)=F_{lj}(a_f)\thinspace S_l(x;a_g): a spatial factor SlS_l that depends on the geometry aga_g, and a source factor FljF_{lj} that depends on the remaining inputs afa_f. Two examples:

  • Pipe flow. F0F_0 is Stokes flow in the channel. The spatial factors are the responses of the discretized Stokes operator of each shape to smooth sine sources χk=sin⁡(mπξ/10)sin⁡(nπη)\chi_k=\sin(m\pi\xi/10)\sin(n\pi\eta), m≤32m\le32, n≤4n\le4, with (ξ,η)(\xi,\eta) the mesh coordinates along and across the channel, placed in the interior rows of the vorticity equation; the kk-th column is the horizontal velocity of that response, scaled by its RMS in a straight channel (nb=128n_b=128). There is no source factor (F≡1F\equiv1), and the coefficients are learned from the nine shape coordinates. All columns share the matrix of the prior solve, so they cost one factorization per shape.

  • Annulus. The unit disk is split by four interface radii into a central disk and four concentric rings. The prior imposes continuity of the field and its flux at the interfaces; the reference adds an interface thermal resistance. The source factors are the Fourier coefficients g^ms\hat g^{s}_m of the boundary data g(θ)g(\theta) on ρ=1\rho=1, with (ρ,θ)(\rho,\theta) polar coordinates, m≤12m\le12, s∈{cos⁡,sin⁡}s\in\lbrace \cos,\sin\rbrace. The spatial factors are separable, Pms(θ) Zml(ρ;ag)P_{ms}(\theta)\thinspace Z_{ml}(\rho;a_g), with Pms(θ)=s(mθ)P_{ms}(\theta)=s(m\theta) and, in each region, the two radial solutions (ρ/rj+1)m(\rho/r_{j+1})^m and (rj/ρ)m(r_j/\rho)^m (only the first in the central disk; r1,…,r4r_1,\dots,r_4 are the interface radii, r0=0r_0=0, r5=1r_5=1), nine radial functions in all (nb=2⋅12⋅9=216n_b=2\cdot12\cdot9=216). The coefficients are learned from the four interface radii.

We will add a table to Sec. 3 that lists SlS_l, FljF_{lj} and nbn_b for every benchmark, following the forms already given in App. B.8.

Recommendations

Reviewer’s comment: Recommendations. Give more general recipes for injecting equation priors into other architectures (attention-based, non-Fourier bases), and extend the experiments to larger real-world problems.

AuthorsOpenReview ↗

Response to Reviewer VYxh (2/2)

(continued: Recommendations)

Response.

A general recipe and other architectures. We followed this recommendation; the recipe and the attention-based results are in the general comment. On non-Fourier bases: none of the three backbones uses one (DeepONet learns its trunk, GNOT attends over input tokens, Transolver learns a soft partition into slices), and the fixed bases are Green responses, Laurent series or transfer-matrix modes, depending on the equation.

Larger real-world problems. We agree that our benchmarks are small, though several use realistic geometry (pipe flow, airfoils, a plate with a hole). Weather forecasting is autoregressive, where an approximate model enters as a solver-in-the-loop correction (response to ycrY, W6-P1). We will name large-scale and time-dependent problems as the main future direction in Sec. 5.

Writing and minor notes

Reviewer’s comment: Writing. Several terms and phrases read as AI-generated and obscure Sections 2–3; the minor notes ask why a lower residual need not mean a more accurate solution, and for examples of direct and indirect priors.

Response.

We thank the reviewer for the specific examples. As stated in our AI-use disclosure, English is not our first language, and we used language models to draft and translate parts of the text. We checked every passage ourselves and judged it accurate and academically appropriate, but the reviewer’s examples show that this judgment was not sufficient: some phrasing that read naturally to us obscures the meaning for a reader. They point to real problems of clarity, and we will rewrite Sections 2 and 3 in a plainer academic register, defining each term when it first appears. For the cases listed:

Current textRevision
“Learner”“the neural operator” in the introduction; “the corrector” where it refers to εθ\varepsilon_\theta (App. H, K)
“Instance-side factor”“the input-dependent factor ψ(a)\psi(a) produced by the branch” (for the indirect prior, the per-instance numbers q(a)(1,α)Tq(a)(1,\alpha)^{\mathsf T} fixed by the equation)
“raw operator”“the backbone trained on data alone, without equation information (RAW)”
“For a direct prior this residual is not an arbitrary leftover.”“When F0F_0 is linear and invertible and uu, upu_p share boundary conditions, the residual u−up=−L0−1Δ(u)u-u_p=-L_0^{-1}\Delta(u) is fixed by the terms that F0F_0 omits, through the inverse of its operator (Lemma 1).”
“Two qualifications.”Removed in both places (Sec. 4.2, App. C.6); the caveats are stated in full sentences.
“What the rank-two prior removes.”“Reference pressure p⋆p^\star (top), rank-two prior upu_p (middle) and residual p⋆−upp^\star-u_p (bottom) on three held-out cases; shared color scale.”
“The gains are conditional.” (Sec. 5)“The gains depend on how much of the solution the prior resolves, on the backbone, on whether the basis spans the residual, and on the data available to estimate its coefficients (Secs. 4.1–4.3).”

Minor note 1. We will expand the sentence, with the reference already cited there (Chen et al., 2026, arXiv:2606.06164). A residual loss is evaluated at a finite set of points, and the loss can admit fields that are not solutions but have a lower residual than the true solution. Chen et al. give a direct example with the exact equation: on their ice-melting benchmark, one optimizer drives the residual sixteen times below another yet ends with a higher error, because a diffused interface has smaller gradients, and hence a smaller residual, than a sharp interface; the trained model’s residual is even two orders of magnitude below that of the reference solution (their App. I). For time-dependent problems, a small residual averaged over the time domain can also hide a solution that violates causality (their Sec. 3). We will keep this separate from the misspecification bias of an approximate F0F_0 (Proposition 1), which is what our 3D layered results illustrate.

Minor note 2. We will add examples at the start of Sec. 2. Direct priors (17 of our 18 benchmarks) return the field from F0F_0 without labels, either by a numerical solve (the reaction, heat and battery benchmarks, and a Stokes solve standing in for Navier–Stokes on Pipe) or by a classical analytic construction (a spectral expansion, a transfer matrix for layered media, an effective-medium coefficient for heterogeneous media, or a complex potential in plane elasticity). The indirect prior (AirfRANS) returns only the per-instance factor q(a)(1,α)Tq(a)(1,\alpha)^{\mathsf T} from the momentum equation; its spatial profile is fitted by least squares on the same NN training labels.

Reviewer_ycrYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks where a known, but not exact, equation should enter a neural operator when labeled solutions are few. The authors split that choice into three levels. Level 1 adds the equation residual to the training loss. Level 2 solves a simpler equation first and trains the network only on the leftover field. Level 3 also gives the leftover a basis taken from that simpler equation, and the network learns coefficients in that basis.

They build a custom prior for eighteen problems. Six of those also get a residual basis. The bases come from spectral modes, transfer matrices, a Stokes solve, a complex potential, and thin-airfoil scaling. The backbones are DeepONet and, on some tasks, FNO. They compare the prior with a raw operator and with a physics-informed loss at its best loss weight. They also compare the equation basis with a random basis and with a POD basis of the same width.

Critical strengths

The question is clear, and the three levels are easy to tell apart. Equation (1), Table 1, and Figure 1 match.

The paper does not hide the losses. On Darcy with FNO, the prior does not beat the raw model (Table 2: 0.0193 versus 0.0185). On Pipe with FNO, the prior is worse (0.0196 versus 0.0169). On optical, the equation basis does not separate from the prior or from POD (Table 3, p ≥ 0.6). On Pipe at N ≥ 500, POD passes the equation basis. On elasticity, the basis does not win at N = 10. Section 5 states that the gain tracks how much of the field the prior already explains.

The controls are the right kind. PRIOR-RAND and PRIOR-POD match the width and the parameter count. Appendix J also runs POD-DeepONet (Lu et al., 2022). Appendix H swaps the anchor for a retrieved field (Yue et al., 2025), a POD field, a mean field, and a learned DeepONet (Ma et al., 2026a). Five seeds and a spread are reported. The λ choice is called an oracle, and Table 8 checks a leave-one-seed-out rule.

Proposition 1 is a useful warning. If the simple equation is wrong, a residual penalty moves the target, and more labels do not undo that move. Lemma 1 says what the leftover field is when the simple operator is linear. The authors mark where those identities do not apply.

Appendix B is concrete. A reader can see the Stokes prior, the effective-medium Darcy prior, and the airfoil factor q(1, α). The AirfRANS case also separates dynamic-pressure scaling (CP-NET) from the fitted rank-two field (PRIOR) and from both together (CP-PRIOR).

Critical weaknesses

1. The big wins are mostly a hand-built solver plus a small coefficient fit.

On 2D Poisson, the prior error is already 0.1827, PRIOR reaches 0.0161, and PRIOR-STRUCT reaches 0.0025 (Table 3). Appendix F says that dropping the free corrector leaves 258 parameters and is about five times better than PRIOR at N = 200. The target coefficients do not depend on the instance (Appendix B.8). So this run is a sine series plus a short coefficient vector. It is not evidence that a neural operator learned the map.

On 3D layered, the prior alone has error 0.1751. PRIOR, which adds a generic corrector, is worse, at 0.2377. PRIOR-STRUCT reaches 0.0347. Table 17 says the free corrector changes that error by about 1.01 at N = 200. The transfer-matrix basis carries the result. The same pattern holds on the annulus (0.6649 to 0.0251).

This idea is old in another form. Lagaris, Likas, and Fotiadis (1998) write the trial field so that known conditions sit outside the network, and the network fits only the rest. POD-DeepONet (Lu et al., 2022) fixes the trunk to data modes. Operator inference learns coefficients in a chosen basis (Peherstorfer and Willcox, 2016). The new part here is a catalog of equation-specific bases. There is no procedure that builds the prior from a general F0. Section 5 says the constructions are specific on purpose. Then the method is a set of eighteen derivations, not one operator that a new equation can use.

2. The prior helps most when the backbone is weak, or when the prior is already close to the answer.

On 1D nonlinear reaction, the prior alone has error 0.0029. PRIOR reaches 0.0004 under DeepONet. The physics-informed loss stays at 0.0230, next to the raw error 0.0246 (Table 2). Starting from a field that is already within 0.3% will beat a loss that still has to learn the whole field. That cell should not lead the claim.

The paper’s own split says the same thing. Below the median prior error, the prior is ahead by about 6.48 times. Above it, the two uses are about even (0.99 times). FNO, which already fits these grids well, is the even case. Table 11 also shows the prior losing to the raw FNO on Darcy-type tasks at small N. The abstract says the prior compares favorably with the loss. The fair sentence is the one in Section 5: it helps in proportion to how much it already resolves, and a strong backbone can remove the gap.

3. The physics-informed baseline is a single loss weight, not the methods that paper cites as the standard.

Level 1 is a data loss plus λ times an F0 residual (Equation 4). λ is picked on the test set. That choice favors the loss, and the authors say so. It is still a weak physics-informed method. PI-DeepONet (Wang et al., 2021) and PINO (Li et al., 2024) are the operator versions of this idea, and both are cited, but neither training recipe is run. PINO uses a function-space residual. The gradient-balance fixes in Wang, Yu, and Perdikaris (2021) and the causal weights in Wang, Sankaran, and Perdikaris (2022) are not tried. On 3D layered the best λ is 1, at the edge of the grid, and the physics-informed error stays near 1.03 while the raw error is about 1.61 (Table 11). A stuck loss does not show that every residual penalty is weak. It shows that this penalty did not train.

A hybrid is also missing. The natural model is the prior plus a corrector, trained with both the data loss and a residual loss. Level 1 and Level 2 are only compared as separate slots. Proposition 1 says they can fight when F0 is wrong. That fight should be measured, not only argued.

4. Several “no loss” tasks cannot host a loss because of the chosen output.

Table 7 says Pipe has no matched residual because the network outputs one velocity component, so pressure and the other velocity are missing. Section 4.3 says AirfRANS has no matched residual because the network outputs pressure on a fixed Cartesian mask, not the full RANS state on the body mesh. Those are output choices. A model that emits the fields the equation needs could still use a residual. The sentence “the equation is more useful before training” is not identified on those tasks. It is identified only on the eleven tasks that do admit a residual, and even there the margin depends on the backbone.

5. Public benchmarks are not run in their public form, and stronger operators are absent.

Darcy and Burgers from Li et al. (2021), Pipe from Li et al. (2023), and AirfRANS from Bonnet et al. (2022) are public. The AirfRANS task here is pressure on points that lie outside every airfoil, with N ≤ 500. That is not the official AirfRANS field or force task, so the numbers cannot be read against that benchmark.

GNOT is cited in the introduction (Hao et al., 2023) and never trained. Geo-FNO is the Pipe model in Li et al. (2023) and is not a backbone. Transolver is a strong geometry operator on this same family (Wu et al., 2024) and is absent. The DeepONet used here is a width-50 network taken from a reference setup (Appendix C.2). On 3D layered it has 73,350 parameters and the raw error stays above 1. A small network makes a hand-built basis look stronger. The matched comparison inside one code is fair. The claim about neural operators in general needs one stronger backbone on Pipe, Darcy, and the layered task.

6. Low-fidelity operator learning is the closest line, and the main experiments do not use it.

Meng and Karniadakis (2020) already learn a correction on top of a cheap model. Howard et al. (2023) do this inside DeepONet with more than one fidelity. Kochkov et al. (2021) and Um et al. (2020) learn a correction around a solver step. Those papers are cited or sit next to the cited ones, but the experiments do not include a multifidelity DeepONet or a solver-in-the-loop corrector. Appendix H does put a learned DeepONet in the anchor slot. That is useful. It is not the same as a low-fidelity solver that is itself a neural operator trained on cheap data.

The Darcy prior is a harmonic or geometric mean (Appendix B.7). That is a standard effective-medium step, and the leftover error stays near 0.19 to 0.25 (Table 9). A two-scale homogenization corrector would be the classical next term. It is not built, and Level 3 is skipped for every Darcy task. Those are exactly the tasks where the prior loses or ties.

The elasticity basis is a Laurent series from a conformal map (Appendix B.4). That construction comes from classical complex-potential elasticity (Muskhelishvili, 1953). The optical prior is a product of 2 × 2 layer matrices (Appendix B.5), which is the standard characteristic-matrix method (Yeh, 1988). The airfoil factor is dynamic pressure times a linear function of angle (Equation 19), which is thin-airfoil scaling (Anderson, 2017). These are good priors. They should be named as known solutions. On optical, POD-DeepONet at 16 modes reaches 0.1049, better than PRIOR-STRUCT at 0.1258 (Table 21). Where the borrowed basis is only a sensitivity, a data basis wins.

7. Per-query cost is not the training cost.

Appendix L says building the prior for the test set costs about 0.05% of one PRIOR training run. Training is paid once. At test time, Pipe still needs a Stokes solve on a 129 × 129 grid for every new shape (Appendix B.6), and elasticity still needs its map and series. The surrogate’s job, as the introduction states, is to avoid a new high-fidelity solve. A reader needs the time of the full solver, the time of F0 plus the network, and the time of the network alone. The Darcy speedups (239 times and 628 times) are reported. Pipe, the annulus, and the layered closed form are not given in that same per-query form.

Questions for authors
  1. On 2D Poisson, can you report a least-squares fit of the known modal coefficients, with no neural trunk? Appendix F already says 258 parameters beat the full prior model. If that fit matches PRIOR-STRUCT, the operator is not what produces the error of 0.0025.

  2. Can you add one hybrid on the eleven tasks that admit a residual: start from up, train the corrector with the data loss, and add the same oracle residual penalty? Please do this where the prior is already very close (1D nonlinear reaction, prior error 0.0029) and where it is weak (Darcy). Proposition 1 says the penalty can pull the field back toward the wrong target. The table should show that pull.

  3. Can you rerun Level 1 with the PINO residual (Li et al., 2024) and with one gradient-balance rule (Wang, Yu, and Perdikaris, 2021)? The current physics-informed model is one scalar λ on a supervised loss. On 3D layered it stays near error 1. A stuck run is a weak base for the sentence that a prior beats a loss.

  4. For Pipe, can you emit streamwise velocity, transverse velocity, and pressure, and then form the Navier–Stokes residual? Table 7 says the loss is unavailable because the output is one component. That is a choice. The same question applies to AirfRANS if the model emits velocity and eddy viscosity on the body mesh, which Section 4.3 already says is possible in principle.

  5. Please split the geometric-mean claim by backbone. How many FNO cells have PRIOR ahead of PI, and what is the FNO-only ratio? Table 2 and Table 11 suggest the DeepONet cells drive the 2.73 times figure.

  6. On Pipe and Darcy (Li et al., 2021), can you replace the width-50 DeepONet with Geo-FNO (Li et al., 2023) or the FNO you already use, and report PRIOR against RAW at N = 200 and N = 1000? Please also try one stronger geometry model, such as GNOT (Hao et al., 2023), on the layered and annulus tasks, where the raw DeepONet error stays large.

  7. Please add a plain multifidelity DeepONet baseline (Howard et al., 2023) that sees the same up as an input channel, without a custom basis. This separates “give the network the cheap field” from “replace the trunk by an analytic basis.”

  8. For AirfRANS, please state in the main text that the grid, the pressure-only target, and the N ≤ 500 pool are not the official AirfRANS split (Bonnet et al., 2022). CP-NET is already within noise of PRIOR at N = 200 (1.76 times versus 1.69 times, p = 0.33). Can you say that the fitted profile c(x) is the weak part, and that q-scaling is the strong part?

  9. Please give per-query time for the full solver, for F0 plus the network, and for the network alone, on Pipe, elasticity, annulus, and 3D layered. Appendix L currently compares the prior build with training time.

  10. The layered prior gets worse as N grows: 0.1908 at N = 10 and 0.2641 at N = 1000, while up alone stays at 0.1751 (Table 11). Why does more data make the corrector worse? Is this fixed by a larger width, a smaller learning rate, or by stopping on a validation split? There is no validation split now (Appendix C.2).

References

  • Anderson (2017). Fundamentals of Aerodynamics. McGraw-Hill.
  • Bonnet et al. (2022). AirfRANS. NeurIPS Datasets and Benchmarks.
  • Hao et al. (2023). GNOT. ICML.
  • Howard, Perego, Karniadakis, and Stinis (2023). Multifidelity deep operator networks. Journal of Computational Physics.
  • Kochkov et al. (2021). Machine learning–accelerated computational fluid dynamics. PNAS.
  • Lagaris, Likas, and Fotiadis (1998). Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Networks.
  • Li et al. (2021). Fourier neural operator. ICLR.
  • Li et al. (2023). Geo-FNO. JMLR.
  • Li et al. (2024). Physics-informed neural operator. ACM/IMS Journal of Data Science.
  • Lu et al. (2021). DeepONet. Nature Machine Intelligence.
  • Lu et al. (2022). POD-DeepONet comparison. Computer Methods in Applied Mechanics and Engineering.
  • Ma et al. (2026a). Physics-guided correction under model misspecification.
  • Meng and Karniadakis (2020). A composite neural network that learns from multi-fidelity data. Journal of Computational Physics.
  • Muskhelishvili (1953). Some Basic Problems of the Mathematical Theory of Elasticity. Noordhoff.
  • Peherstorfer and Willcox (2016). Data-driven operator inference for nonintrusive projection-based model reduction. Computer Methods in Applied Mechanics and Engineering.
  • Um et al. (2020). Solver-in-the-loop. NeurIPS.
  • Wang, Sankaran, and Perdikaris (2022). Respecting causality is all you need for training physics-informed neural networks.
  • Wang, Wang, and Perdikaris (2021). Learning the solution operator of parametric PDEs with physics-informed DeepONets. Science Advances.
  • Wang, Yu, and Perdikaris (2021). Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing.
  • Wu et al. (2024). Transolver. ICML.
  • Yeh (1988). Optical Waves in Layered Media. Wiley.
  • Yue et al. (2025). DeltaPhi. NeurIPS.
11 comments
AuthorsOpenReview ↗

Summary of our response to Reviewer ycrY

We thank the reviewer for the detailed review and ran the requested experiments, except a full-state residual on AirfRANS, causal weights and a homogenization corrector, for the reasons given below. The large gains come from classical solutions; building on them rather than relearning them is the intent of the method.

ConcernResponsePart
W1/Q1: hand-built solver + small fit2D Poisson conceded; elsewhere the basis does most of the work and POD is far worse (R6–R8)1–2/6
W2/Q2: weak backbones; hybridAbstract will be revised; hybrid shows the predicted pull to u_p (R9)2/6
W3/Q3: weak PI baselinePINO, gradient balance: PRIOR lower in 15/19 (R10)2–3/6
W4/Q4/Q5: “no loss” tasks; by backboneClaim scoped to 11 benchmarks; Pipe residual does not help (R11)3–4/6
W5/Q6/Q8: public form, backbones, AirfRANSAnchor neutral under FNO (R12); profile and q-scaling complementary (R13)4/6
W6/Q7: multifidelity, known solutionsu_p as input does not reproduce the basis gain on 5/6 (R14)4–5/6
W7/Q9: per-query costTimings (R15); speed claim will be limited5–6/6
Q10: layered prior worsens with NCorrector overfits6/6
AuthorsOpenReview ↗

Response to Reviewer ycrY (1/6)

Several points (W1, W1-P3, W6-P3) observe that the large gains come from classical solutions rather than the network. We agree, and this is the intent: where such knowledge exists, a neural operator should build on it rather than relearn it from labels or replace it by a data basis such as POD.

W1, Q1

Reviewer’s comment: W1-P1 and Q1. On 2D Poisson the result is a sine series plus a short coefficient vector, not evidence of operator learning; please report a least-squares fit of the modal coefficients with no neural trunk.

Response to W1-P1 and Q1.

We ran the requested fit. Table R6 compares a least-squares fit of the modal coefficients (no network) with our models on 2D Poisson.

ModelError
upu_p alone0.1827
PRIOR (upu_p + corrector)0.0161
Least squares on the equation basis (no network)0.00318
PRIOR-STRUCT without corrector (258 params)0.0033
PRIOR-STRUCT0.0025±0.00010.0025 \pm 0.0001
POD of residuals + upu_p, ridge regression0.054
POD of residuals + upu_p, neural branch (PRIOR-POD)0.0134±0.00040.0134 \pm 0.0004
POD of solutions, ridge regression0.357
POD of solutions, neural branch (POD-DeepONet, best pp)0.178

Table R6. 2D Poisson, N=200N=200, relative L2L^2 error (mean ±\pm s.d. over five seeds). Least squares equals the oracle projection onto the retained modes and is 0.00318 at every NN; PRIOR-STRUCT goes from 0.0162 (N=10N=10) to 0.0023 (N=1000N=1000).

The reviewer is right on 2D Poisson: least squares with no network matches the model, so this benchmark is not evidence of operator learning, and we will say so in Sec. 4. The fit is easy only because of the basis: the equation says which source coefficient drives which mode, so each mode needs one constant. A POD basis mixes the modes and is far worse with the same regression, although it spans the residual better.

Reviewer’s comment: W1-P2. On 3D layered and annulus the transfer-matrix basis carries the result, and the free corrector changes the error by about 1%.

Response to W1-P2.

We agree. Here the coefficients depend on the geometry, so we applied the Q1 test with a linear or quadratic ridge map in aga_g (Table R7). Least squares on the equation basis is about as good as PRIOR-STRUCT at N=200N=200; the neural map overtakes it at N=1000N=1000 on annulus and Pipe, and elasticity, whose coupling is hardest, levels at N=1000N=1000.

BenchmarkN=200N=200 u-LS lin.N=200N=200 u-LS quad.N=200N=200 STRUCTN=200N=200 ratioN=1000N=1000 u-LS lin.N=1000N=1000 u-LS quad.N=1000N=1000 STRUCTN=1000N=1000 ratio
annulus0.04300.02630.02511.050.04170.02510.02011.25
elasticity0.06980.04570.06260.730.06170.03670.03491.05
Pipe0.02960.02170.02440.890.02910.01860.01251.49
3D layered$^\dagger$0.0458–0.03471.320.0390–0.03021.29

Table R7. Least squares on the equation-derived basis (u-LS) versus PRIOR-STRUCT, relative L2L^2 error (mean over five seeds). u-LS maps aga_g to the coefficients with a linear (lin.) or quadratic (quad.) ridge regression; the ridge weight is chosen by cross-validation on the training set. Ratio: u-LS (quad.) / STRUCT; <1<1: least squares is better.

†^\daggerLinear only; the quadratic map has about 46,000 coefficients. The ratio uses the linear map.

With the regression, prior and samples fixed and only the basis changed (Table R8), the equation basis wins on all six benchmarks (p<10−4p<10^{-4}), by the widest margin where it fixes how the coefficients depend on the input (geometry on annulus and 3D layered, mode coupling on 2D Poisson).

BenchmarkEQPODRANDPOD/EQRAND/EQ
annulus0.0430±0.0014\mathbf{0.0430\pm0.0014}0.515±0.0050.515\pm0.0050.665±0.0010.665\pm0.00112.015.5
elasticity0.0698±0.0023\mathbf{0.0698\pm0.0023}0.093±0.0010.093\pm0.0010.319±0.0000.319\pm0.0001.334.57
Pipe0.0296±0.0003\mathbf{0.0296\pm0.0003}0.035±0.0000.035\pm0.0000.107±0.0000.107\pm0.0001.183.61
optical0.2059±0.0001\mathbf{0.2059\pm0.0001}0.250±0.0020.250\pm0.0020.290±0.0010.290\pm0.0011.221.41
2D Poisson0.0032±0.0000\mathbf{0.0032\pm0.0000}0.054±0.0010.054\pm0.0010.183±0.0000.183\pm0.00017.157.5
3D layered0.0458±0.0019\mathbf{0.0458\pm0.0019}0.172±0.0030.172\pm0.0030.181±0.0010.181\pm0.0013.763.95

Table R8. Same regression, different basis: least squares on top of upu_p, N=200N=200, relative L2L^2 error (mean ±\pm s.d. over five seeds). EQ: the equation-derived basis with the linear map of Table R7. POD: the top POD modes of the training residuals u−upu-u_p, with as many modes as the equation basis allows; for each benchmark we report the better of two coefficient maps (the same map as EQ, or a ridge map from all instance inputs to the POD coefficients), which favours POD. RAND: random orthonormal basis of the same width.

AuthorsOpenReview ↗

Response to Reviewer ycrY (2/6)

(continued: W1, Q1)

The learned components still matter: the neural map overtakes least squares with more data, the corrector carries the result when the basis is incomplete (optical, App. F), and one setting serves all benchmarks where least squares needs a hand-chosen form for each.

Reviewer’s comment: W1-P3. The idea is old in other forms (Lagaris et al., POD-DeepONet, operator inference); the method is a catalog of equation-specific derivations with no procedure that builds the prior from a general F0F_0.

Response to W1-P3.

We agree that each ingredient has precedents, and we do not propose a general procedure for building a prior from an arbitrary F0F_0. The contribution is the question: where should a known approximate equation enter a neural operator, as a loss, a prior or a representation?

Relation to the cited work. Lagaris et al. (1998, Eq. 4) place known boundary values outside the network for a single instance; Lu et al. (2022, Sec. 3.1.3) carry this into DeepONet. Our prior instead removes what F0F_0 determines in the interior, across instances. Lu et al. (2022, Sec. 3.1.2) feed analytic features to a learned trunk; we fix the basis itself. POD-DeepONet fixes the trunk to modes of the same NN labels; our basis also encodes how the coefficients depend on the input, and with the same regression POD is worse on all six benchmarks (Table R8). Operator inference learns reduced dynamics in a POD basis; the input-to-coefficient surrogate we learn is the case Peherstorfer and Willcox set aside (their Sec. 1).

Generality. The constructions are equation-specific (Sec. 5), but all fill the same slots of Eq. 10 and carry over to GNOT and Transolver (general comment). Automatic construction from a general F0F_0 is future work.

Revisions.

We will call 2D Poisson a degenerate case, add Tables R6–R8 to Sec. 4 and App. F, extend the related work, and add the backbone results to a new appendix.

W2, Q2

Reviewer’s comment: W2 and Q2. The prior helps most when the backbone is weak or the prior is already close, so the abstract overstates it; please add a hybrid (prior, corrector, data loss and residual loss) to measure the pull predicted by Proposition 1.

Response to W2 and Q2.

Claim. We agree and will replace the abstract sentence by the Sec. 5 statement: as an anchor, the prior helps in proportion to how much it resolves, and a strong backbone can close the gap. 1D nonlinear reaction will no longer lead the claim.

What the split misses. It measures the anchor (Level 2). The basis (Level 3) helps most where the prior is far off: on annulus and 3D layered the anchor barely helps, yet the basis lowers the error several-fold (Table 3), because it supplies how the solution depends on the instance. We will state both.

The hybrid (Q2). u^=up+εθ\hat u=u_p+\varepsilon_\theta, trained with the data loss and the PI residual, same oracle grid, budget and seeds, on all 19 pairs. Proposition 1 predicts that F0F_0’s residual is minimized at upu_p, so a larger λ\lambda should pull the model from PRIOR to upu_p.

λ\lambdaLi Darcy, FNO (prior weak) ErrorLi Darcy, FNO (prior weak) DistanceLi Darcy, FNO (prior weak) Residual1D nonlinear reaction, FNO (prior close) Error1D nonlinear reaction, FNO (prior close) Distance
10−610^{-6}0.01930.2471.470.00060.0031
0.20.20.06970.2160.370.00060.0031
110.17240.1763.4×10−33.4\times10^{-3}0.00120.0029
10310^{3}0.25050.0272.0×10−52.0\times10^{-5}0.00290.0002
EndpointsPRIOR 0.0193, upu_p alone 0.2511PRIOR 0.0006, upu_p alone 0.0029

Table R9. The pull of the residual penalty, N=200N=200, mean over five seeds. Error: relative L2L^2 error. Distance: ∥u^−up∥/∥up∥\Vert \hat u-u_p\Vert /\Vert u_p\Vert. Last row: the two endpoints.

Evidence. The pull is as predicted (Table R9): the residual falls, the prediction moves onto upu_p and the error rises to that of upu_p. It costs little where the prior is close and undoes the data where it is weak, and more data does not help (on Li Darcy the gap widens with NN). The oracle therefore switches the penalty off almost everywhere.

Revisions.

Abstract and lead result as above; the split for both uses of F0F_0 and the hybrid in Sec. 4 and App. C.

W3, Q3

Reviewer’s comment: W3 and Q3. The physics-informed baseline is a single loss weight; please rerun Level 1 with the PINO residual and a gradient-balance rule, since on 3D layered the current loss does not train.

Response to W3 and Q3.

Setup. On all 19 pairs we added (i) the PINO loss of Li et al. (2024), a function-space residual with 1,000 extra unlabeled inputs and λ\lambda picked on the test set, and (ii) the learning-rate annealing rule of Wang, Teng and Perdikaris (2021, Alg. 2.1), which needs no oracle:

Reviewer_5GzKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a unified view of equation-derived information for neural operators under approximate physical operators and scarce high-fidelity data in three levels: training constraints, priors, and residual structures. In summary, level 1 information is the common physics-informed residual loss used for training neural operators. Conversely, level 2 information (prior anchor) is derived before training either by running the approximate physical operator for all inputs (direct) or by analysis of the operator, which yields closed-form low-dimensional factors along with spatial profiles fitted via least squares on the training labels (indirect). The fixed (untrainable) prior is added to a parameterized residual, which can be further decomposed into a structured residual (level 3, applied to DeepONet) and a corrector neural operator. The structured residual combines an untrainable equation-derived basis (capturing both linear source terms and geometric factors) with a trainable coefficient network that takes only the geometric parameters as input. Finally, the structured residual and the corrector are trained in stages to maximize the portion of the total residual realized by the span of the structured residual. The authors argue that level 2 and 3 information, which are derived from the approximate equations prior to training, are more effective and computationally cheaper than the common level 1 physics-informed training while avoiding the target bias caused by misspecified physics losses.

The authors evaluate this progression across 18 PDE benchmarks under varying data budgets. On benchmarks admitting a residual loss, level 2 priors generally outperform oracle-weighted level 1 physics-informed losses, achieving large gains when the prior resolves most of the solution, while performing comparably when the prior is coarse (such as effective-medium Darcy priors). On the six benchmarks that expose a residual basis, level 3 structural injection significantly improves over capacity-matched random and POD basis controls on five of the six problems. Finally, on the AirfRANS benchmark where no matched residual loss can be formed, combining an indirect prior with dynamic-pressure scaling consistently outperforms both raw operators and data-derived baselines.

Critical strengths
  • The unified view into the equation-derived information is very interesting, and the authors make convincing theoretical and empirical arguments against the common residual loss training.
  • The experiments are very well-designed and rigorous. The selected benchmarks cover a broad range of PDEs, ablation studies are thorough and insightful, and the results are transparently reported.
  • When the approximate equation resolves a meaningful portion of the solution or exposes the right residual basis, levels 2 and 3 yield substantial accuracy gains in the low-data regime.
Critical weaknesses

Overall, the novel framework and the strong results outweigh the weaknesses for me. However, the weaknesses below prevent me from recommending clear acceptance.

  1. Limited practical scalability. Unlike data-driven methods and even regular physics-informed residual training, level 2 and 3 information require manual analysis of the equations and domain expertise to derive the basis functions, transfer matrices, etc. Scalability to complex problems, unstructured geometric representations, and 3D industrial problems, where high-fidelity numerical simulations are most expensive, remains unclear.
  2. Inference costs. For direct priors and structured residuals, inference requires running the approximate solver (and constructing the instance-dependent basis) for each new test sample. This is acceptable when approximate models are orders of magnitude faster than high-fidelity solvers. However, the cost of the approximate solve, along with the forward pass of the network, may not always be justifiable compared to running the full numerical solver, e.g., for elasticity, where the approximate solver is only 2.2×2.2\times faster than the full reference solver (Appendix L).
  3. Architecture limitations. The full three-level framework is primarily limited to DeepONet. While Level 2 gains under FNO are smaller or absent on several benchmarks, level 3 information is only tested with DeepONet, and more recent transformer-based neural operators are not considered at all. Furthermore, Table 17 shows that when the basis spans the residual, the corrector ϵθ\epsilon_\theta is mostly redundant (removing it eliminates a large share of the parameters while the error changes by at most 4% on 3D layered, elasticity and Pipe, and improves slightly on annulus), meaning Level 3 is doing low-dimensional basis-coefficient regression rather than deep operator correction. At this point, the framework can be seen as a classic reduced-order model with no corrector network. Meanwhile, on Pipe at N=200N=200, a raw FNO (0.01690.0169, Table 2) already outperforms DeepONet with Level 3 structural injection (0.02440.0244, Table 3). Recent transformer-based neural operators may provide further raw improvements, especially for geometric domains
Questions for authors
  1. What prevents the corrector ϵθ\epsilon_\theta in Level 3 from being any black-box neural operator (FNO or transformer-based)?

  2. Have the authors systematically ablated the severity of approximation within a single PDE? I.e. what determines the point where even equation priors or bases harm purely data-driven learning?

6 comments
AuthorsOpenReview ↗

Summary of our response to Reviewer 5GzK

We thank the reviewer for the positive assessment and the precise questions.

ConcernResponsePart
W1: manual derivation, scalabilityPriors and bases can be built numerically from a solver; injection runs on point clouds. Large 3D problems will be stated as a limitation1/2
W2: inference costPer-query timings (R15). Prior pays only where the reference is harder than F0F_0; speed claim will be limited1/2
W3: DeepONet only; "a reduced-order model"GNOT/Transolver results (general comment). Partly a ROM, by design: the basis comes from the equation, varies per instance, costs no labels; POD with the same regression is worse on all six benchmarks. Agreed on Pipe1/2
Q1: any neural operator as corrector?Yes; Table 3 conclusions hold with GNOT, Transolver and FNO (R3–R4)1–2/2
Q2: severity of the approximationκ\kappa sweep (R5): the anchor fails when the prior error is as large as the solution; the basis keeps capturing 96–99% of the residual2/2
AuthorsOpenReview ↗

Response to Reviewer 5GzK (1/2)

W1

Reviewer’s comment: W1. Levels 2 and 3 need manual analysis and domain expertise to derive bases and transfer matrices, so it is unclear how they scale to complex, unstructured and 3D industrial problems.

Response to W1.

We agree that Levels 2 and 3 need a solvable approximate model, and we offer no automatic way to derive one; we will state this as the main limitation in Sec. 5. Two points narrow it.

  • Closed forms are not required. Several priors are plain numerical solves (1D heat, battery, reaction; a Stokes solve on Pipe), and bases can be built numerically too: Stokes responses to sine sources on Pipe, finite-difference sensitivities on optical. The manual step is choosing F0F_0, often an existing coarse solver, not deriving a basis.

  • Unstructured geometry. The injection runs on point-cloud operators (GNOT, Transolver; general comment).

Large 3D problems remain untested: our 3D tasks are small, and scaling the construction to industrial problems is future work.

W2

Reviewer’s comment: W2. Inference needs an approximate solve (and a basis) for every test sample, which may not pay off against the full solver; elasticity is only 2.2$\times$ faster (App. L).

Response to W2.

We agree. Per-query timings are in Table R15 (response to ycrY, W7). The prior pays only where the reference is a harder problem than F0F_0 (Darcy, Pipe); on annulus and 3D layered F0F_0 costs as much as the reference, and on elasticity the shape-dependent basis makes the structured model slower than the reference. We will limit the speed claim accordingly and present the others as accuracy results. The 2.2× in App. L came from an earlier timing run; under the single protocol of Table R15 the elasticity construction is 2.0× cheaper than the reference and the full query 1.9×.

W3

Reviewer’s comment: W3. Level 3 is tested only with DeepONet; when the basis spans the residual the corrector is redundant, so Level 3 reduces to a reduced-order model, and a raw FNO already beats it on Pipe.

Response to W3.

Transformer-based operators. See the general comment and Q1 below: with GNOT or Transolver the basis keeps its advantage, and the raw transformer errors stay large on annulus and 3D layered, so a stronger backbone alone does not solve these tasks.

A reduced-order model? In part, yes, and we see this as the point rather than a weakness. When the basis spans the residual, most of the work is a low-dimensional coefficient regression; our response to Reviewer ycrY (W1-P2, Tables R7 and R8) shows that least squares on the same basis comes close at N=200N=200. It differs from a classic reduced-order model in where the basis comes from. A classic model takes a fixed basis from snapshots, as POD does; ours is derived from the equation, changes with each instance and costs no labels. With the same regression, prior and samples, a POD basis is worse on all six benchmarks. The network still matters in three places: the neural coefficient map overtakes least squares with more data, the corrector carries the result when the basis is incomplete (optical), and one setting serves all benchmarks where least squares needs a hand-chosen form for each.

Pipe. We agree. Pipe is the benchmark where one data-derived basis already describes most instances, so the equation basis brings little, and a raw FNO is the best model there; the transformer correctors with the basis do not beat it either. We will state this in Sec. 4.

Q1

Reviewer’s comment: Q1. What prevents the Level 3 corrector from being any black-box neural operator (FNO or transformer-based)?

The same results also answer Reviewer ycrY, Q6 and W5 (a stronger geometry model such as GNOT on the layered and annulus tasks; Transolver absent).

Answer.

Nothing in the formulation does. Eq. 12 only asks the corrector to map (a,x)(a,x) to a correction field, so we keep the fixed-basis term and its coefficient network exactly as in the paper and replace only the learned part εθ\varepsilon_\theta:

u^(x;a)=up(x;a)+∑kβθ,k(a) ΦF0,k(x)+εbackbone(x;a).(E4)\displaystyle \hat u(x;a)=u_p(x;a)+\sum_{k}\beta_{\theta,k}(a)\thinspace \Phi_{\mathcal F_0,k}(x) +\varepsilon_{\text{backbone}}(x;a). \qquad\text{(E4)}

This needs no backbone-specific design. RAND and POD replace Φ\Phi in the same slot, and PRIOR drops the fixed-basis term, as in Table 3. The one practical requirement is that the corrector accepts the query points of the benchmark: FNO needs a regular grid, so it applies only to 2D Poisson and Pipe, while GNOT and Transolver run on all six benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 5GzK (2/2)

(continued: Q1)

BenchmarkDeepONet PRIORDeepONet STRUCTGNOT PRIORGNOT STRUCTGNOT RANDGNOT PODTransolver PRIORTransolver STRUCTTransolver RANDTransolver POD
annulus0.51950.02510.38000.04140.41790.46080.34330.03730.33820.4331
elasticity0.10980.06260.12940.05380.14240.16540.09130.05370.10500.1597
2D Poisson0.01610.00250.10850.00320.12060.01880.04250.00320.03500.0227
3D layered0.23770.03470.17530.03050.18950.16870.24240.03080.25000.2131
Pipe0.04340.02440.08090.02810.08140.02940.08450.03900.08050.0290
optical0.12290.12580.07970.08670.09200.08540.05060.13470.07110.0577

Table R3. Other correctors (Eq. (E4)): the paper’s fixed-basis term with GNOT or Transolver as the corrector, N=200N=200, relative L2L^2 error (mean over five seeds). DeepONet columns are Table 3 of the paper. PRIOR: upu_p + corrector. STRUCT / RAND / POD: equation-derived, random, or POD basis in the fixed-basis slot.

BenchmarkRAWPRIORSTRUCT
2D Poisson0.01220.00400.0021
Pipe0.01690.01960.0285

Table R4. The fixed-basis term of Eq. (E4) with FNO as the corrector, on the two benchmarks with a grid, N=200N=200, relative L2L^2 error (mean over five seeds). RAW: FNO alone.

Reading Tables R3 and R4.

Changing the corrector does not change the conclusion of Table 3. On the four benchmarks where the equation basis captures the residual (annulus, elasticity, 2D Poisson, 3D layered), STRUCT is far below PRIOR and below the random and POD bases with every corrector, and DeepONet, GNOT and Transolver reach similar errors. On Pipe a POD basis is as good with GNOT and better with Transolver (0.0290 against 0.0390), and with FNO as the corrector the basis makes the error worse (0.0285 against 0.0169). On optical, where the basis is only a first-order sensitivity, the fixed term does not help, and with Transolver it even hurts.

The intuition is that once the basis is fixed, the corrector only has to model what lies outside its span, so its architecture matters little. The result is set by how much of the residual the basis captures, which is the point of ycrY W1-P2. FNO fits this reading on 2D Poisson but not on Pipe, where raw FNO remains the best model, as your W3 notes. On Pipe a strong grid model already captures most of the residual, which plausibly leaves little for the basis to add.

Q2

Reviewer’s comment: Q2. Has the severity of the approximation been ablated within one PDE, i.e., at what point do equation priors or bases start to harm data-driven learning?

Response to Q2.

We ran this ablation on annulus, where the missing physics is known exactly. The reference adds an interface thermal resistance that the prior omits; we scale this resistance by κ\kappa, regenerate the data, and keep the prior and the basis unchanged, so κ\kappa sets how wrong F0F_0 is (κ=1\kappa=1 is the paper’s setting).

κ\kappaupu_p aloneOutside spanRAWPRIORSTRUCTRAND
0.0010.0493.8%0.3110.0560.00310.055
0.010.2132.6%0.3450.2310.01100.234
0.10.4421.7%0.4440.4020.01710.414
10.6651.4%0.5770.5200.02510.557
101.2731.3%1.0501.0850.08451.149

Table R5. Severity of the approximation on annulus, N=200N=200, relative L2L^2 error (mean over five seeds). “Outside span”: share of the residual u−upu-u_p that the basis cannot represent.

The two uses of F0F_0 break down at different points (Table R5). The prior as an anchor is sensitive to how large its error is: it stays close to upu_p alone at every κ\kappa, and once upu_p is off by about 100% (κ=10\kappa=10) it no longer beats RAW. The basis is sensitive to whether its structure is right, not to the size of the error: it captures 96–99% of the residual at every κ\kappa, because a stronger interface term changes only the coefficients, and STRUCT stays 12–100$\times$ below RAW with no crossover in this range. A random basis of the same width does no better than PRIOR, so the gain comes from the structure of the basis, not from its size.

The answer to the reviewer’s question is therefore: an anchor stops helping when the prior is about as far from the solution as the solution itself, while a basis stops helping when it no longer spans the residual, which, in this ablation, a larger error of the same kind does not cause. We will add this study to Sec. 4.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Overview of the rebuttal: concerns and the experiments that answer them

We thank the three reviewers for their detailed and constructive reviews. We are encouraged that they found the unified view of equation-derived information interesting and the arguments against residual-loss training convincing (5GzK), the question clear and the three levels easy to tell apart (ycrY), the experiments rigorous, thorough and applied to a wide range of benchmarks (5GzK, VYxh), the controls of the right kind and the losses reported transparently (ycrY, 5GzK), and Proposition 1 a useful warning about misspecified residuals (ycrY).

Our message. Mathematical physics has accumulated, over more than a century, solutions that say how a field depends on geometry, loading and material: spectral expansions, transfer matrices, complex potentials, Stokes responses. Where such knowledge exists, a neural operator should not have to rediscover it from scarce labels, nor replace it by a basis extracted from the data, as POD does. Our aim is to let the network build on what is already known and learn only what is not. The new experiments support this: the equation-derived basis wins most clearly where it encodes how the solution changes from one instance to the next, which data alone would have to learn.

The common thread of the reviews is whether the framework generalizes beyond DeepONet and whether the gains survive stronger baselines and backbones. We ran every requested experiment except three, for the reasons given in the responses: a full-state residual on AirfRANS, causal weights (all our residual benchmarks are steady), and a homogenization corrector for Darcy. All new results are labelled Tables R1–R15 and appear in the comments listed below. Paper equations, tables and appendices keep their numbers.

Concern (reviewers)What we didResultWhere
Is Level 3 tied to DeepONet's trunk/branch structure? (all three)Rewrote structural injection as learned basis × coefficients + fixed basis × coefficients and built it into GNOT and Transolver; also kept it as a separate term with GNOT/Transolver/FNO as the correctorAs a separate term, the transformer correctors reproduce the DeepONet results of Table 3. Inside every layer, the equation basis beats random and POD bases of the same width on four of six benchmarks in GNOT and two in TransolverGeneral comment; R1–R4
Is the prior-versus-loss comparison fair? (ycrY, 5GzK)PINO and gradient-balanced residuals; prior + residual hybrid; split by backbone; Pipe with full-state output and a Navier–Stokes residualStronger residual recipes do not change the conclusion. The hybrid shows the predicted pull toward the approximate equation. The prior's margin is large under DeepONet and about even under FNO, so we will scope the claim and revise the abstractycrY W2–W4; R9–R11
Is the gain from the network or from a hand-built basis? (ycrY, 5GzK)Least squares on the equation basis; same regression with a POD basis; multifidelity DeepONets that see upu_p as an inputThe basis does most of the work and the network adds with more data. A POD basis with the same regression is worse on all six benchmarks. Seeing upu_p as an input does not reproduce the basis gainycrY W1, W6; R6–R8, R14
How wrong can F0F_0 be? (5GzK)Scaled the omitted physics on annulusThe anchor fails once the prior error is about as large as the solution; the basis keeps working as long as it spans the residual5GzK Q2; R5
Stronger backbones, public benchmarks (ycrY)FNO and the official Geo-FNO on Pipe and Li Darcy; GNOT and Transolver on layered and annulus; AirfRANS ablationWith strong grid backbones the anchor is roughly neutral; the raw transformer errors stay large where the basis helps; the AirfRANS profile and the qq-scaling are complementaryycrY W5; R12–R13
Inference cost (5GzK, ycrY)Per-query time of the reference solver, F0F_0 + network, and network aloneThe prior pays where the reference is a harder problem than F0F_0 (Darcy, Pipe), not on annulus and 3D layered; we will limit the speed claimycrY W7; R15
Clarity and terminology (VYxh)Revision of Sections 2–3 and each flagged phraseVYxh

Where we agree with the reviewers and will revise the paper. 2D Poisson is a degenerate case, not evidence of operator learning. A strong backbone can close the gap between prior and loss, and the abstract will say so. On optical the sensitivity basis is weak and a multifidelity model is the better use of F0F_0. On Pipe a raw FNO is the best model. The speed claim holds only where the reference is a harder problem. The 3D layered failure of the residual loss is an optimisation failure, and we will correct its explanation in App. C.3.

AuthorsOpenReview ↗

General response: structural injection inside attention-based backbones (1/3)

All three reviewers raise the same underlying question: does the structure of Level 3 apply only to DeepONet, or is it a framework that carries over to other neural operators? Reviewer VYxh asks this directly in Q1 and the main weakness (“Does the structural injection only apply to DeepONet? It seems like it relies on the trunk/branch structure”; more general recipes for attention-based architectures), Reviewer 5GzK in W3 (“level 3 information is only tested with DeepONet”), and Reviewer ycrY in Q6 and W5 (GNOT and Transolver as stronger geometry models).

To answer it, we rewrite what the paper shows only for DeepONet in the language of transformers. Eq. 12 is restated without the trunk/branch split, and the equation-derived basis is built into every layer of GNOT and Transolver (Steps 1–2, Tables R1–R2). Our response to Reviewer 5GzK (Q1) covers the simpler case, where Eq. 12 stays a separate term next to the backbone.

This shows that the framework extends to other families of neural operators. We do not claim that these variants outperform the original backbones. How well they do depends on the characteristics of each architecture and on how it models the interaction between the basis and the PDE coefficients, which we leave to future work.

Notation.

x1,…,xNx_1,\dots,x_N are the query points, y1,…,yT2y_1,\dots,y_{T_2} the input tokens (GNOT), and upu_p the prior solution. S~∈RN×K\tilde S\in\mathbb R^{N\times K} is the spatial factor of Φ\Phi with unit-RMS columns, S~k=Sk/rms(Sk)\tilde S_k=S_k/\mathrm{rms}(S_k); columns that vanish on the query points of an instance are masked (valid set). The least-squares projection is

S~†H=(G+δI)−1S~THN,G=S~TS~N,δ=10−6 tr⁡GK.\displaystyle \tilde S^{\dagger}H=(G+\delta I)^{-1}\frac{\tilde S^{{\mathsf T}}H}{N},\qquad G=\frac{\tilde S^{{\mathsf T}}\tilde S}{N},\qquad \delta=10^{-6}\thinspace \frac{\operatorname{tr}G}{K}.

Starting point.

Structural injection (Eq. 12) is a DeepONet whose trunk is split into a fixed, equation-derived part and a learned part:

u^θ(x;a)=up(x;a)+∑k=1nbβθ,k(a)⏟coefficient ΦF0,k(x)⏟fixed basis+∑i=1pbθ,i(a)⏟coefficient tθ,i(x)⏟learned basis⏟εθ(x;a).(E1)\displaystyle \hat u_\theta(x;a) =u_p(x;a)+\sum_{k=1}^{n_b}\underbrace{\beta_{\theta,k}(a)}_{\text{coefficient}}\thinspace \underbrace{\Phi_{\mathcal F_0,k}(x)}_{\text{fixed basis}} +\underbrace{\sum_{i=1}^{p}\underbrace{b_{\theta,i}(a)}_{\text{coefficient}}\thinspace \underbrace{t_{\theta,i}(x)}_{\text{learned basis}}}_{\varepsilon_\theta(x;a)} . \qquad\text{(E1)}

Inserting Φ\Phi into every layer.

Eq. (E4) treats Eq. (E1) as a term. Here we insert Φ\Phi inside every layer of the backbone, with no separate head (B0B_0; B0B_0+F for Transolver). Steps 1–2 derive the form this takes in each backbone.

Step 1: vector-valued coefficients.

Lift every scalar coefficient to a vector in RC\mathbb R^{C} and mix the channels with an output projection WoW_o:

out(x)=Wo[∑iti(x) bi(a)+∑kΦk(x) mk(a)],bi, mk∈RC.(E2)\displaystyle \mathrm{out}(x)=W_o\Big[\sum_{i} t_i(x)\thinspace b_i(a)+\sum_{k}\Phi_k(x)\thinspace m_k(a)\Big], \qquad b_i,\thinspace m_k\in\mathbb R^{C}. \qquad\text{(E2)}

With a linear read-out wow_o, woTmk=βkw_o^{\mathsf T} m_k=\beta_k and woTbi=bθ,iw_o^{\mathsf T} b_i=b_{\theta,i}, Eq. (E2) reduces to the sum in Eq. (E1). Inside a deep backbone, where nonlinear layers follow, Eq. (E2) is the feature-level analogue of Eq. (E1), applied at every layer.

Step 2: coefficients as weighted sums.

Every coefficient is a weighted sum of value vectors,

ck(a)=∑jωkj vj,(E3)\displaystyle c_k(a)=\sum_{j}\omega_{kj}\thinspace v_j , \qquad\text{(E3)}

and the backbones differ in where the values come from and how the weights ω\omega are formed:

GNOT:ωkj=1T2(WkΦyj)k,vj=WvΦyj(input tokens, j=1,…,T2)\displaystyle \text{GNOT:}\quad \omega_{kj}=\tfrac1{T_2}\big(W^{\Phi}_k y_j\big)_k,\quad v_j=W^{\Phi}_v y_j\quad\text{(input tokens, } j=1,\dots,T_2)

Transolver:ωki=[(G+δI)−1S~T/N]ki,vi=fi(hidden states at the query points, i=1,…,N)\displaystyle \text{Transolver:}\quad \omega_{ki}=\big[(G+\delta I)^{-1}\tilde S^{\mathsf T}/N\big]_{ki},\quad v_i=f_i\quad\text{(hidden states at the query points, } i=1,\dots,N)

In GNOT the coefficients are read directly from the input; in Transolver the input reaches them only through the hidden state ff, which is why B0B_0+F adds a direct input term (T3$'$).

AuthorsOpenReview ↗

General response: structural injection inside attention-based backbones (2/3)

GNOT

G1: learned basis == softmax query.

t(x)=q~(x)=softmax(Wqh(x)),b=∑j=1T2k~jvjT,k~j=softmax(Wkyj), vj=Wvyj.\displaystyle t(x)=\tilde q(x)=\mathrm{softmax}\big(W_q h(x)\big),\qquad b=\sum_{j=1}^{T_2} \tilde k_j v_j^{\mathsf T},\quad \tilde k_j=\mathrm{softmax}(W_k y_j),~v_j=W_v y_j .

G2: replace 1/T21/T_2 by a query-wise normalisation (stability).

∑iti bi⟶At=q~t+q~tT∑jk~jvjTq~t⋅∑jk~j.\displaystyle \sum_i t_i\thinspace b_i\enspace \longrightarrow\enspace A_t=\tilde q_t+\frac{\tilde q_t^{\mathsf T}\sum_j \tilde k_j v_j^{\mathsf T}} {\tilde q_t\cdot\sum_j \tilde k_j}.

(The residual term q~t\tilde q_t is part of the original GNOT layer.)

G3: fixed basis =S~=\tilde S, without normalisation.

Since S~\tilde S changes sign, the denominator of G2 could vanish, so the plain average of Eq. (E3) is kept:

∑kΦk mk⟶Φt=∑kS~k(xt) mk,M=1T2∑j=1T2(WkΦyj)(WvΦyj) ⁣T∈RK×C,\displaystyle \sum_k\Phi_k\thinspace m_k\enspace \longrightarrow\enspace \Phi_t=\sum_k \tilde S_k(x_t)\thinspace m_k,\qquad M=\frac1{T_2}\sum_{j=1}^{T_2} \big(W^{\Phi}_k y_j\big)\big(W^{\Phi}_v y_j\big)^{\negthinspace {\mathsf T}}\in\mathbb R^{K\times C},

where mkm_k is the kk-th row of MM.

outt=Wo[At+gℓ Φt]\displaystyle \boxed{\enspace \mathrm{out}_t=W_o\big[A_t+g_\ell\thinspace \Phi_t\big]\enspace }

Transolver

T1: learned basis == soft domain partition (slices).

fi=Wfxhi,tg(xi)=wig=softmaxg(WsWxhi/τ).\displaystyle f_i=W_{fx}h_i,\qquad t_g(x_i)=w_{ig}=\mathrm{softmax}_g\big(W_sW_x h_i/\tau\big).

T2: coefficients == slice averages, then attention among slices.

zg=∑iwigfi∑iwig⟶os=Attn(zs)⟹∑gwig ogs.\displaystyle z_g=\frac{\sum_i w_{ig} f_i}{\sum_i w_{ig}} \enspace \longrightarrow\enspace o^{s}=\mathrm{Attn}(z^{s}) \quad\Longrightarrow\quad \sum_g w_{ig}\thinspace o^{s}_g .

T3: fixed basis =S~=\tilde S; coefficients by projection, then attention over Φ\Phi- and slice-tokens.

zΦ=S~†f+emode+etype⟶oΦ=Attn(qΦ;[kΦ;ks])⟹∑kS~k(xi) okΦ.\displaystyle z^{\Phi}=\tilde S^{\dagger} f+e_{\text{mode}}+e_{\text{type}} \enspace \longrightarrow\enspace o^{\Phi}=\mathrm{Attn}\big(q^{\Phi};[k^{\Phi};k^{s}]\big) \quad\Longrightarrow\quad \sum_k \tilde S_k(x_i)\thinspace o^{\Phi}_k .

emode∈RK×De_{\text{mode}}\in\mathbb R^{K\times D} tells attention which token is which mode, and etypee_{\text{type}} separates Φ\Phi-tokens from slice tokens. The mask is one-way: Φ\Phi-queries attend to [Φ;slice][\Phi;\text{slice}] keys, while slice queries attend to slice keys only, as in the original layer. Invalid modes are removed from the keys and their outputs are set to zero.

T3$'$: B0B_0+F, the input enters the coefficient path directly.

Each spatial column ll of S~\tilde S carries JJ input factors {Flj(af)}j=1J\lbrace F_{lj}(a_f)\rbrace _{j=1}^{J}, i.e. Φ(l,j)=FljSl\Phi_{(l,j)}=F_{lj}S_l (J=1J=1 for annulus and 2D Poisson, 44 for 3D layered, 88 for elasticity; optical and Pipe have F≡1F\equiv1, carry no instance data in FF, and so B0B_0+F =B0=B_0). With F~lj=Flj(af)/rmstrain(Flj)\tilde F_{lj}=F_{lj}(a_f)/\mathrm{rms}_{\text{train}}(F_{lj}), the Φ\Phi-token of mode ll is

zlΦ=[S~†f]l+emode,l+etype+F~lTWF,F~l∈RJ, WF∈RJ×D,\displaystyle z^{\Phi}_l=\big[\tilde S^{\dagger}f\big]_l+e_{\text{mode},l}+e_{\text{type}} +\tilde F_l^{{\mathsf T}}W_F,\qquad \tilde F_l\in\mathbb R^{J},~W_F\in\mathbb R^{J\times D},

with WFW_F shared across heads and initialised to zero, so B0B_0+F starts as B0B_0. The rand and pod controls receive the same F~\tilde F; only S~\tilde S is replaced.

outi=Wo[∑gwig ogs+gℓ∑kS~k(xi) okΦ]\displaystyle \boxed{\enspace \mathrm{out}_i=W_o\Big[\sum_g w_{ig}\thinspace o^{s}_g +g_\ell\sum_k \tilde S_k(x_i)\thinspace o^{\Phi}_k\Big]\enspace }

Model and initialisation (B0B_0, B0B_0+F).

The prediction is u^=up+εS~(a,x)\hat u=u_p+\varepsilon_{\tilde S}(a,x), with the Φ\Phi-path inserted in every layer as in the boxed equations. GNOT uses B0B_0, since its Φ\Phi-coefficients already read the input tokens yjy_j; Transolver uses B0B_0+F. All gates start at gℓ=0g_\ell=0 (WF=0W_F=0 as well), so at initialisation each layer equals the original backbone and the model equals PRIOR (up+εu_p+\varepsilon) for the same seed.

Summary.

All three forms read learned basis ×\times coefficients ++ fixed basis ×\times coefficients; they differ only in how the coefficients are obtained (Eq. (E3)). GNOT forms the fixed-basis coefficients from the input tokens with learned weights, whereas Transolver projects the hidden state onto the basis, adds the input factors F~\tilde F, and lets the resulting coefficients interact through attention.

Remark.

Eq. (E1) uses Φ(l,j)=Flj(af) Sl(x;ag)\Phi_{(l,j)}=F_{lj}(a_f)\thinspace S_l(x;a_g), while the boxed layers use the spatial factor S~\tilde S alone. This loses nothing: since

∑l,jβljFljSl=∑lSl(∑jFljβlj),\displaystyle \sum_{l,j}\beta_{lj}F_{lj}S_l=\sum_l S_l\Big(\sum_j F_{lj}\beta_{lj}\Big),

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.