acceptodds
Under review as a conference paper at ICLR 2027

Paradifferential Neural Operators: Better Call Bony For Non-Linear PDEs

Abstract

Neural Operators have emerged as powerful architectures for learning mappings between infinite-dimensional function spaces. While predominantly deployed as PDE surrogates, their broader promise in data-driven AI for Science requires models capable of rigorously resolving complex non-linear dynamics. To address this, we introduce ParaNO, a Fourier-based neural operator grounded in paradifferential calculus. Crucially, ParaNO leverages Bony's decomposition to efficiently learn the cross-scale frequency interactions caused by non-linearities. Alongside our method, we introduce The Non-Linear Six (The NL6), a curated benchmark designed to systematically isolate distinct classes of algebraic non-linearities without geometric confounding. ParaNO improves over SotA baselines by 23% on Navier–Stokes and reduces errors by factors of 2x to 4.6x on the NL6. Our code, benchmark, and network weights will be made available upon publication.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PDE, neural operator, paradifferential

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_mxbFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new network architecture in the field of Neural Operators, which learn the dynamics of PDEs. The architecture is designed to capture the non-linear responses that appear in the dynamics. Based on the idea that linear operations do not mix frequency components, the authors add a branch to FNO that explicitly mixes frequency components, so that the model can represent strong non-linearity. This branch is based on the Bony decomposition, which is also used in the analysis of non-linear PDEs, so a good fit can be expected.

Strictly speaking, FNO also learns interactions between frequency components implicitly. Still, making this explicit is a meaningful contribution.

Critical strengths

The performance also seems consistently high. However, this is hard to evaluate until the problem below (the test data is used for model selection) is resolved.

Critical weaknesses

(1) The major problem is the handling of the results. The following sentence suggests that the reported performance is chosen by looking at the test data, which is an improper use of the data. At minimum, a fair selection procedure is needed, such as a train–valid–test split, or stopping at a fixed number of iterations matched to prior work without looking at the test set.

  • l310 "we evaluate and compare models in terms of the lowest relative ℓ2 test errors achieved during training"

(2) The motivation is unclear.

  • The abstract says "capable of rigorously resolving complex non-linear dynamics", but the evaluated problems actually include linear dynamics. It is natural that even linear dynamics produce an output that is non-linear with respect to the input, and calling this "non-linear" is a misuse of the term.

In addition, the paper appears to contain many small errors in its descriptions. Below I list at least the ones that are clearly contradictory. Given this many errors, there are probably more that cannot be predicted from the manuscript, as well as ones I have overlooked. At minimum, these must be thoroughly corrected.

(3) In the first half of the paper, the authors acknowledge that FNO performs frequency mixing, but in the second half they deny it. It is true that FNO is less explicit than the proposed method, but this is somewhat of an overclaim.

  • l138 "While this implicit coupling is theoretically universal, it is structurally agnostic to the underlying PDE and the true spectral interactions."
  • l442 "Rather than ignoring cross-frequency mixing (like FNO) or mixing modes uniformly (like HO-FNO), ParaNO groups frequencies into interacting octaves and routes them directly through the distinct algebraic regimes depicted in Fig. 3:"

(4) This work introduces interactions between frequency components. That is fine, but I question calling it an "inductive bias," as in the sentence below. This term is usually used when the learning procedure or the architecture is given a property such as equivariance, or when some implicit regularization arises. A new computational term by itself is generally not called an inductive bias. At minimum, the authors should explain its effect in relation to the properties of the target being learned (for example, that it is strongly related to some specific property of PDEs).

  • l144 "we introduce an inductive bias that explicitly models the frequency interactions and resonant behaviours that characterise the vast majority of governing PDEs."

(5) Also, FNO does not apply a non-linear activation inside its spectral branch either (the activation is applied after the branch), so the following sentence is factually incorrect.

  • l244 "Crucially, unlike FNO, we omit any non-linear activation within this branch,"

(6)

  • Table 1: "all baselines retain their original codebase"
  • l317: "As CATO has not publicly released an official implementation at the time of writing,"

(7)

  • Table 1: "While retrained,"
  • l321: "The sole exception is Plasticity, as we were unable to reproduce the baseline metrics despite extensive efforts, we report the originally published values for this task."

(8)

  • l338: "Across Geo-PDE (Tab. 1), ParaNO excels on Native Grids containing active spectral mixing, improving upon HO-FNO by 23% on Navier–Stokes. However, it underperforms relative to spatial methods (like CATO and MSPT) on Darcy flow and most interpolated geometries."

In fact, according to Table 1, ParaNO is inferior to MSPT only on PIPE.

(9)

  • Abstract: "reduces errors by factors of 2× to 4.6× on the NL6"

In fact, ParaNO is inferior on HYP, and the reduction on VAW is 1.6×.

(10)

  • l53: "we also introduce the Non Linear Six (the NL6) benchmark, a dataset of six non-linear PDEs that exhibit these dynamics, numerically simulated on periodic grids."
  • l278: "While VAW and DNO simulate state evolutions that are technically linear, they represent highly non-linear mappings with respect to their heterogeneous input fields, testing a model's ability to process variable fractional orders and analytic non-polynomials, respectively." If so, they are linear PDEs.
Questions for authors

What is the performance when it does not depend on the test data? If it is comparable to the current manuscript, that would be a strong reason for acceptance.

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer mxbF

We thank the reviewer for their time and efforts invested in our paper. Our detailed responses are given below. As the revised evaluation shows that performance remains comparable when selection is independent of the test data, we hope this directly addresses the reviewer’s stated criterion that such a result would provide "a strong reason for acceptance".

W1) Test-set selection and updated results We agree with the reviewer that our experimental procedure had to be improved. Therefore, we implement both the reviewer's suggestions. on Geo-PDE we now report test metrics at the final training epoch rather than the lowest test errors, and we specify this evaluation protocol in the revised paper. Additionally, we redefine the data splits in The NL6 to have train, test, and validation sets. On The NL6 we perform checkpoint selection on the validation set and report test metrics with the selected checkpoint. The updated evaluation leaves ParaNO’s ranking unchanged; while some relative margins shift under the revised protocol, ParaNO retains substantial leads on all tasks where it ranks first.

W2) Nonlinear dynamics versus interactions We thank the reviewer for pointing out this nuanced distinction. We agree that nonlinear PDE dynamics and nonlinear solution operators are not equivalent concepts. We have replaced "non-linear dynamics" with "non-linear interactions" in the abstract and text. The updated term accurately reflects the multiplicative coupling between fields and spatially varying coefficients that Bony's decomposition is designed to resolve. This clarification does not change our architectural motivation, which is to introduce a paradifferential inductive bias for learning operators governed by cross-scale frequency interactions.

W3) Implicit versus explicit frequency mixing We intended to distinguish implicit activation-driven mixing from explicit scale-structured interactions, and have revised the wording accordingly.

W4) Architectural inductive bias We use "inductive bias" in its standard architectural sense: a structural assumption that biases the learner toward some patterns or representations over others. ParaNO does not merely add a computational term, but imposes Bony’s low–high, high–low, and remainder interactions learned by the network. This mathematically-grounded structure directly reflects the scale organisation of non-linear products in paradifferential calculus, and we now make this explicit by referring to it as an "architectural" inductive bias.

W5) Activation placement in FNO We thank the reviewer for noticing this small phrasing ambiguity. We have clarified that SθS_\theta corresponds to "the pre-activation transform of a FNO layer" and stressed that, "unlike a complete FNO layer, SθS_\theta does not include the subsequent non-linear activation".

W6/W7) Incomplete Table caption We apologise for the omission in the Table's caption, we have now improved its alignment with the main text.

W8) MSPT comparison We have corrected this comparison by removing MSPT from the sentence.

W9) NL6 improvement range We thank the reviewer for noticing this minor discrepancy, which now specifies the improved performances in the correct ranges (which have been updated in accordance with the improved experimental protocol) and limits the claim to five of the six PDEs.

W10) PDE linearity versus task nonlinearity We agree that the terminology was imprecise. We have therefore revised the manuscript to refer to NL6 as six PDE-related learning tasks exhibiting non-linear interactions, broadening the terminology to encompass non-linearities arising in the target input–output map rather than only in the governing PDE.

Dear Reviewer mxbF,

Thank you again for your time and thoughtful review. As the discussion closes soon, November 18 (AoE), we would appreciate hearing whether our rebuttal addresses your concerns. We would be happy to clarify any remaining points before the deadline.

All the Best, The authors

I thank the authors for the substantial revision and response.

(1) I am glad that the authors acknowledged the problem in the original evaluation protocol. However, Tables A3 and A4 still appear to contain the same problematic results as those in the first submission. The same probably holds for the other figures and tables that were not replaced (e.g., Fig. 1)

I cannot agree with the response to (4). The term "architectural inductive bias" sounds even more like CNNs or GNNs. I think a description such as "an architecture inspired by paradifferential calculus" would be reasonable.

The response to (5) appears inconsistent with Eq. (6). I cannot determine what the actual structure is.

In addition, the substantial revision seems to introduce various inconsistencies. For example, Table A5 summarizes seven benchmarks, while its caption says "The five standard benchmark problems."

It was not easy to find the time to read the entire manuscript very carefully within the limited discussion phase, but many errors still seem to remain. Although some of the issues have been fixed, the manuscript does not appear to be ready for publication. Therefore, it is difficult for me to change my rating.

We thank the reviewer for taking the time to go through our revised manusript, we appreciate that time in the discussion phase is limited.

  • We thank the reviewer for noticing this minor mismatch, we have now updated both tables. We would like to clarify that, for computational reasons, Table A3 was obtained by tuning the models on Navier Stokes and then testing them on The NL6. Models now adopt their last training checkpoint following the revised protocol. Note that this did not change the overall outcome of our experiment. Figure 1 has also been updated, but, as the reviewer may appreciate, changes are barely visible because metrics changed only minimally. We also confirm that all other figures and tables are correct and up to date.

  • We would like to clarify our use of the terminology. The concept of inductive bias is considerably broader than equivariance or invariance, which are a particular case of it. Mitchell (1980) defines inductive bias in terms of the assumptions or preferences that guide a learning algorithm towards particular generalizations, which is precisely our case. More specifically, Battaglia et al. (2018, Box 2) explicitly identify architectural structure as a source of inductive bias.

    In ParaNO, the bias does not arise from merely introducing a bilinear operation, nor from Bony's decomposition alone, which is an algebraic identity. Rather, it arises from the structured parametrisation of bilinear spectral interactions: the architecture explicitly distinguishes low–high, high–low, and comparable-scale interactions, assigning learnable responses conditioned on both interaction type and frequency band, while sharing channel-mixing parameters across bands.

    This introduces a preference for scale-structured bilinear operators rather than treating all frequency interactions uniformly. We therefore believe that architectural inductive bias is an appropriate and established description of this design principle, while an architecture inspired by paradifferential calculus accurately describes its mathematical motivation. The two descriptions are complementary rather than mutually exclusive.

  • Eq. 6 is correct and consistent with response number 5. Eq. 6 defines the whole ParaNO layer as:

    h(l+1)=h(l)+σ(cSSθ(h)+cPPθ(u,v))h^{(l+1)} = h^{(l)} + \sigma\left(c_\mathrm{S}\mathcal{S}_\theta(h) + c_\mathrm{P} \mathcal{P}_\theta(u, v)\right)

    Here, the activation function σ\sigma is applied to the linear combination of the linear (Sθ\mathcal{S}_\theta) and non-linear (Pθ\mathcal{P}_\theta) branches. As it can be observed from Eq. 2 of the FNO paper, the activation function is applied to what they define as Wvt(x)+(K(a;ϕ)vt)(x)Wv_t(x) +(\mathcal{K}(a;\phi)v_t)(x), with (K(a;ϕ)vt)(x)(\mathcal{K}(a;\phi)v_t)(x) being defined at their Eq. 4. This is equivalent to our linear branch Sθ\mathcal{S}_\theta. Their Eq. 2 thus corresponds to σ(Sθ)\sigma(\mathcal{S}_\theta)). Sθ\mathcal{S}_\theta is therefore correctly classified as "the pre-activation transform of a FNO layer" and it includes all the FNO transformations except the activation function.

  • Once again, we thank the reviewer for noticing this minor caption inconsistency introduced while updating the table to include two additional rows. We appreciate that even minor inconsistencies like this contribute to the overall quality of the paper, but we are also happy to see that outstanding issues appear related mostly to very minor details and not to fundamental issues with our method or new experimental protocol.

We genuinely hope that our responses, who have now addressed all the reviewer's concerns, can motivate a score increase in alignment with their initially declared intention.

References

  • Mitchell, T. M. (1980). The Need for Biases in Learning Generalizations.
  • Battaglia, P. W., et al. (2018). Relational Inductive Biases, Deep Learning, and Graph Networks.
Reviewer_1WKrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces ParaNO, a neural operator inspired by Bony’s paradifferential calculus. It uses a Littlewood–Paley decomposition to organize interactions between frequency bands and parameterizes them with low-rank kernels. Experiments cover established PDE benchmarks and the newly proposed NL6 suite.

Critical strengths
  1. Interesting mathematically motivated architecture. The use of paradifferential calculus leads to a structured treatment of low–high, high–low, and comparable-scale interactions. This is an appealing inductive bias for operator learning, and the results on NL6 (though self-proposed) suggest that it merits further investigation.

  2. Good parameter efficiency. ParaNO achieves competitive accuracy with approximately 2M parameters. In particular, its reported Navier–Stokes result improves over substantially larger models such as HO-FNO and LaMO, indicating a favorable trade-off between accuracy and parameter count.

Critical weaknesses
  1. Test-set-based selection undermines the evaluation protocol. Section 5 reports the lowest test error achieved during training, and the supplementary code explicitly takes the minimum over the epoch-wise test errors. This uses the test set for model selection and can produce optimistic estimates of generalization. I do not find the appeal to “common practice” persuasive: precedent does not make this an appropriate evaluation protocol. Applying the same procedure to all methods also does not ensure that the selection bias cancels, since their training dynamics may differ. The main comparisons and key ablations should be verified using validation-based checkpoint selection followed by evaluation on an independent test set.

  2. Insufficient positioning and uneven empirical benefits. The related-work discussion would benefit from a more direct comparison of how existing operators model frequency interactions and what ParaNO adds beyond these mechanisms. In particular, characterizing FNO as ignoring cross-frequency mixing is misleading when applied to the full architecture: its spatial nonlinearities already induce such interactions, as the manuscript itself acknowledges. The relevant distinction is therefore explicit, scale-structured interactions versus implicit mixing, rather than the presence or absence of mixing. This distinction should be stated consistently and connected more clearly to the empirical evidence. Although ParaNO performs strongly on NL6 and reports promising Navier–Stokes and Plasticity results, it underperforms the strongest compared methods on Darcy, Airfoil, and Pipe. Together with its substantially higher training cost on Navier–Stokes, these mixed results leave me uncertain about when the proposed structure offers a compelling practical advantage.

Questions for authors
  1. Do the main improvements and key ablation conclusions hold when checkpoints are selected using a separate validation set, followed by independent test evaluation? Please also confirm whether all retrained Navier–Stokes baselines use exactly the same trajectory split as ParaNO.

  2. Could the authors clarify what ParaNO’s explicit, scale-structured interactions add beyond the implicit frequency mixing already present in FNO? In particular, which controlled comparisons isolate this benefit, and what evidence explains when it outweighs the additional computational cost, given the mixed results on established benchmarks?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer 1WKr

We appreciate the reviewer’s recognition of ParaNO’s mathematical motivation and favorable parameter count. We believe the revised evaluation and clarification of its scale-structured interactions directly address the main concerns and support a stronger assessment.

W1) Test-set selection and revised evaluation We completely agree with the reviewer and we rectified our oversight. In particular, on Geo-PDE we now report test metrics at the final training epoch rather than the lowest test errors, and we specify this evaluation protocol in the revised paper. Additionally, we redefine the data splits in The NL6 to have train, test, and validation sets. On The NL6 we perform checkpoint selection on the validation set and report test metrics with the selected checkpoint. The updated evaluation leaves ParaNO’s ranking unchanged; while some relative margins shift under the revised protocol, ParaNO retains substantial leads on all tasks where it ranks first.

W2) Frequency mixing and practical advantage The FNO frequency treatment outlined by the reviewer is correct, we better clarify when mixing is implicit and explicit. We also add Sec. A1 to provide an in-depth description of how frequencies are handled in HO-FNO compared to ParaNO. In addition, since our linear branch follows the standard FNO formulation, we further clarify differences between FNO and ParaNO in Sec. 3 (after Eq. 5). While we acknowledge ParaNO's higher training cost, which we report explicitly in Table A3, the inductive bias introduced through paradifferential calculus substantially improves performance on PDEs with non-linear interactions. Conversely, as ParaNO is not specifically tailored to strongly smoothing input–output maps or irregular geometries, a slight drop in performance can be expected. We therefore highlight both more efficient band-wise interaction construction and extensions to complex geometries as important directions for future work.

Q1) Ablation protocol and shared data splits We are glad to confirm that key ablation conclusions are unaltered as checkpoints were selected from the last training epoch for the reduced-budget ablations. More importanly, we also confirm that all re-trained baselines use the same data splits.

Q2) Controlled evidence and training-cost trade-off We agree that FNO already induces frequency interactions through its pointwise non-linear activations. ParaNO differs by making these interactions explicit and organising them according to relative scale into low–high, high–low, and comparable-scale regimes. Our most direct comparison is FNO$^{(+)}$ in Tables 2 and 3: it retains FNO’s implicit activation-driven mixing while incorporating pre-norm and residual connections, and is evaluated under the same 2M parameter budget. ParaNO substantially outperforms it on all NL6 tasks and on the two newly introduced Well benchmarks. We did not rerun the original FNO on Geo-FNO because it was not among the most competitive available neural operators and, under our fixed computational budget, we prioritised stronger baselines; we nevertheless include FNO$^{(+)}$ on NL6 and The Well because of FNO’s foundational importance and the value of directly comparing against its implicit mixing mechanism.

We acknowledge that ParaNO has a higher training cost and have not established whether these rankings persist at matched wall-clock cost; this overhead is reported in Table A3, and improving the efficiency of the band-wise interaction construction is now highlighted as future work. Our hypothesis is that the additional scale-structured interactions are most useful when cross-scale interactions remain important in the target input–output map, and the strong results on NL6, Navier–Stokes, and The Well provide consistent empirical support for this hypothesis.

I thank the authors for their response. Though the updated draft is better than the original submission, it requires nontrivial work to be considered as a solid submission. I'm retaining my score.

We thank the reviewer for their continued engagement. The significant and non-trivial revisions described in our response are already incorporated into the manuscript, with changes marked in blue. Substantial work has already been completed to address all major concerns, incorporating the reviewer's useful suggestions. As we wonder whether the extent of these revisions may have been overlooked, we briefly highlight the completed work below. We kindly ask the reviewer to consider these changes in their final evaluation and to precisely identify what remains missing to support the assessment that "nontrivial work" is still required. We remain more than happy to address additional concerns throughout the discussion period.

1) Revised evaluation and results. Section 5 now specifies final-epoch evaluation for Geo-PDE and validation-selected checkpoints with held-out testing for NL6. NL6 additionally includes validation-based hyperparameter tuning (Sec. A.2; Tables A1–A2). As confirmed above, retrained baselines share the same data splits, and reduced-budget ablations used final-epoch checkpoints.

Below, errors are ×10−3\times 10^{-3}, with standard deviations omitted for brevity; bold marks the lower error.

Geo-PDE — before (Table 1)

MethodNSDarcyPlasticityAirfoilPipe
ParaNO23.874.900.395.705.45
Best baseline31.113.870.504.163.98

Geo-PDE — after (Table 1)

MethodNSDarcyPlasticityAirfoilPipe
ParaNO23.914.920.395.715.45
Best baseline31.253.870.504.173.98

NL6 — before (original Table 2)

MethodDNOHYPKSSQGSTFVAW
ParaNO1.793.367.614.138.481.02
Best baseline4.952.8415.6619.1214.481.66

NL6 — after (revised Table 3)

MethodDNOHYPKSSQGSTFVAW
ParaNO1.363.036.232.838.540.72
Best baseline4.102.3010.0810.6013.311.56

Thus, correcting the evaluation protocol leaves ParaNO's conclusions unchanged: it retains its wins on Navier–Stokes, Plasticity, and five of six NL6 tasks. We additionally evaluated ParaNO on two benchmarks from The Well, where it outperforms all evaluated baselines on both tasks.

2) Positioning, controlled evidence, and computational cost. Sections 3 and 6 now explicitly distinguish FNO's implicit activation-driven frequency mixing from ParaNO's explicit, scale-structured interactions, while new Sec. A.1 provides a detailed comparison with HO-FNO. Tables 2–3 include FNO(+) under the same 2M-parameter budget, directly addressing the request for a controlled comparison against FNO-style implicit mixing. We also substantially expanded the empirical analysis and updated Fig. 5, Fig. A2, and Figs. A5–A13. Finally, ParaNO's additional computational cost is explicitly reported in Table A4 (50.7 h versus HO-FNO's 18.3 h), and we do not claim superiority under matched wall-clock cost.

We also appreciate that the reviewer assessed the paper positively on presentation (3: "clear and easy to follow") and informativeness (2: "I learned something interesting or useful"), and explicitly identified the mathematically motivated architecture and parameter efficiency as strengths. The remaining concern was therefore principally soundness, for which the review raised the two specific issues above. Since both have now been addressed through substantial modifications and additional experiments, we would greatly appreciate a concrete indication of what soundness issue remains unresolved and what additional work the reviewer believes is necessary. We are fully willing to continue improving the manuscript throughout the remainder of the discussion period.

Reviewer_xjbwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new neural network architecture for neural operators which takes inspiration from Bony's decomposition & paradifferential calculus. Additionally, the paper comes with a benchmark of 6 datasets of non-linear PDEs. This collection, according to the authors, fills a gap between collections that are too simple (PDEBench, APEBench, etc.) or too complex (The Well) to isolate how well a model handles the non-linear part of a PDE.

On existing PDE Problems (NS, Darcy Flow, some mesh-based PDEs interpolated to a regular grid) the performance of the model is decent, but not great. On the PDE datasets designed by the authors, the model is best on 5/6 datasets, and second on one.

Overall, I think it is a well-designed paper with nice connections from theory and experiments, well written with helpful figures. I'm somewhat worried about the practical usefulness and overall potential for impact (detailed below).

Critical strengths
  • The paper overall reads naturally (it is clearly written by humans) and is generally carefully constructed. Figure 2 is very nice and intuitive. Figure 3 (esp. in conjunction with Eq. 1) is also very nice.
  • The connection to Bony's decomposition is quite clear and apt.
  • Comparison across seeds is nice and rarely done in this domain.
  • The method is to my knowledge novel, and, while it seems to be a simple modification to the existing design of FNOs, simple changes can often be the most impactful.
Critical weaknesses

My concerns are largely on the side of experiments, especially on comparing to baselines and overall impact of the submission.

  • There seems to be no explicit discussion on the training details of baselines, so I am assuming that they are trained the same way as ParaNO (as described in §5 Implementation Details). I understand that authors cannot spend excessive amounts of time with tuning hyperparameters of the baseline, but considering how sensitive models can be to learning rate, at least LR should be tuned somewhat. I would be happy to raise my score if this is done, and conclusions hold.
  • In the introduction, the authors argue that their benchmark suite is designed to verify how well models deal with the non-linear parts of PDEs which reads primarily as a hypothesis testing dataset collection. This makes me think that the problems are too stylized to serve as a proxy for performance on real world problems. The authors don't benchmark on more "modern" & complex PDE problems, such as those described in The Well, FlowBench, etc. which seem to have more practical relevance. Moreover, on the problems from GeoPDE, ParaNO's performance is good but not remarkably so.
Questions for authors
  1. (Connecting to the final weakness stated above) While isolating non-linear performance is interesting from an academic perspective, aren't we ultimately interested in how these architectures perform on problems of practical relevance? I'd like to understand whether the authors believe that comparisons on more complex physics are not necessary because ParaNO is not designed for downstream/domain usage, or simply whether they believe that the current results on GeoPDE are sufficient for this.
  2. Where do the authors see the impact of this paper? In which problems or in which domains do they believe ParaNO will have significant impact? Where/when should a domain researcher use ParaNO over another NO architecture?

I will raise my score if questions/weaknesses are sufficiently addressed.

As there is no field for remarks that would not influence my score, I will state some questions and remarks that I believe could help improve the paper below:

  • (minor) In Figure 1 there seems to be a weak correlation between error and parameter size of a model. Does this indicate overfitting? Or do the authors attribute this result to something else?
  • (minor) I find interpolating mesh-based methods to a regular grid when comparing against a method designed specifically for meshes (Transolver) a bit unfair. In particular, per the authors remarks in the paper, interpolation is not favourable for ParaNO anyways.
  • (minor) I find Figure 5 relatively hard to read, at least in part due to the various scales involved in the error plots and GT plots.
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer xjbw [Part1]

We thank the reviewer for their positive assessment of the paper’s clarity, novelty, and analysis. We believe the remaining questions and weaknesses have now been sufficiently addressed and that this warrants a score increase, consistent with the reviewer’s earlier indication.

W1) Baseline tuning and updated results As detailed in Sec. 5 at line 316 of the submitted manuscript (381 of the revised version), "Baseline experiments on Geo-PDE retain the original codebase, hyperparameters, and parameter count of each model". Since we agree with the reviewer that models can be particularly sensitive to hyperparameters' choices, we redefined our experimental protocol on The NL6. In particular, we split the data in a train, validation, and test set. For each model and PDE in the NL6, we then run a hyperparameter search with a Tree-structured Parzen Estimator sampler. We tune batch size, learning rate, and any other key model-specific hyperparameter. Specifically, we tune latent tokens for LaMO, the presence of a physics loss for CATO, as well as modes and order for HO-FNO. For ParaNO we only tune batch size, learning rate, and number of modes in its non-linear branch. All hyperparameters are tuned on the validation set and results reported on a separate test set. Total number of epochs and parameter counts are the same across models and PDEs. We have updated our manuscript with a full description of our updated protocol and updated results accordingly. ParaNO maintains its rankings: first on five of the six NL6 tasks and second on Hyperelasticity. The margins remain substantial, ranging from 1.56×1.56\times to 3.7×3.7\times.

W2) The Well and benchmark relevance Following the reviewer's suggestion of benchmarking our method on more recent datasets, we decided to compare the performance of our ParaNO against LaMO, CATO, HO-FNO, and FNO$^{(+)}$ on Shear Flow and Rayleigh-Bénard from The Well. These PDEs are simulated on native grids and describe complex fluid phenomena. For computational reasons, we downsampled their grids to 128×32128\times32 and 64×12864\times128, and standardised their protocol to ours, thus training for 500 epochs with the objective of predicting 10 timeframes given the previous 10, using a relative L2 loss function. Being unable to run another hyperparameter tuning, we decided to use the hyperparameters selected by each method on Navier--Stokes, and use them to retrain on the new data. Checkpoints are selected on The Well’s validation split and evaluated on its held-out test split. We are happy to report that ParaNO outperforms the other methods, reporting the lowest error on both benchmarks, 13% below the next-best model (CATO) on Rayleigh–Bénard and 22% on shear flow, with every ParaNO seed beating every baseline seed (Table 2).

We would also like to note that our newly introduced benchmarks were deliberately designed to isolate nonlinear interactions between frequency scales, allowing us to test the inductive bias underlying ParaNO in a controlled setting, where we achieve the best results on all but one of the nonlinear PDE benchmarks we introduce. We consider The NL6 to be a very informative benchmark to complement, rather than replace, other benchmarks. Importantly, ParaNO also achieves SOTA performance on Navier–Stokes and Plasticity in GeoPDE, despite the latter involving irregular geometries for which our architecture is not specifically designed. We also obtain the third-best result on Darcy, despite its linear governing equation, which is not the primary motivation for our paradifferential inductive bias. Our strong performance on Plasticity is particularly encouraging, as extending ParaNO to handle irregular geometries natively remains future work.

AuthorsOpenReview ↗

Official Response to Reviewer xjbw [Part2]

Q1) Physical relevance of NL6 While we introduced The NL6 as a benchmark for testing the performance of neural operators on non-linear PDEs over regular grids, it is worth noting that the selected equations govern real-world phenomena with non-negligible practical relevance. In particular, surface quasi-geostrophic turbulence (SQG) is being used to understand the spectrum of motions in the atmosphere and in the ocean. Kuramoto-Sivashinsky equations (KG) are used to model phenomena like flame propagation, reaction–diffusion systems, and plasmas. Shear-thinning flow (STF) models the most common type of non-Newtonian behavior of fluids, with blood also falling in this category. Viscoacoustic wave propagation (VAW) describes the propagation of acoustic waves in viscous media, and finds aplication in seismic science. The Dirichlet--Neumann operator (DNO) plays an important role in the theory of inverse problems and is used here in the free-surface water waves formulation. Hyperelasticity (HYP) describes the non-linear stress-strain behaviours in rubbers, polymers, and biological tissues. We are confident that our superiority in Navier-Stokes, Plasticity, 5 of the NL6 PDEs, as well as the two newly introduced PDEs from the Well prove ParaNO’s empirical benefits beyond the controlled NL6.

Q2) Intended applications and limitations We believe that ParaNO will have a significant impact in AI for Science and hypothesize that ParaNO's inductive bias is most useful when cross-scale interactions remain important in the target input–output map. We further clarified this in the introduction. Its practical value should be weighed against its higher training cost and its current restriction to regular grids. We identify computational efficiency and native support for complex geometries as future research directions.

M1) Model size and overfitting Figure 1 compares different architectures and does not, by itself, establish whether their performance differences arise from overfitting. ParaNO achieves lower test error with fewer parameters, but this observation does not isolate the effects of model size from architectural and training differences.

M2) Interpolated geometries We include these established geometric benchmarks to show the limitations of our current grid-based implementation. Support for irregular geometries remains future work.

M3) Figure 5 readability We retain the current visualization, as using a common scale made several error maps effectively indistinguishable. The overlaid relative ℓ2\ell^2 error provides a consistent quantitative comparison across tasks and models.

The authors largely addressed my concerns by adding better baseline tuning and covering more datasets. I also commend them for rectifying the dataset contamination issues the other reviewers pointed out which I missed initially.

Overall I can see the model and the datasets being useful for the community and for other domains. I have adjusted my score accordingly.

Reviewer_anUYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ParaNO, a Fourier-based neural operator that constructs low–high, high–low, and comparable-scale bilinear interactions using a Littlewood–Paley decomposition. Learned kernels depend on interaction type, dyadic band, and output frequency, with a shared factorization controlling parameter count. A separate linear branch is combined with the nonlinear branch within residual layers.

The authors also introduce NL6, a suite of six periodic-grid tasks intended to probe diverse nonlinear mappings while reducing geometric variation. ParaNO improves the reported Navier–Stokes error by approximately 23% over HO-FNO and performs best on five of six NL6 tasks, while showing weaker performance on several smoothing or interpolated-geometry benchmarks.

Critical strengths
  1. The motivation goes beyond the generic claim that nonlinear PDEs are difficult. It identifies a mismatch between the organization of nonlinear spectral interactions and the uniform treatment of modes in existing constructions. Distinguishing scale modulation from comparable-scale interactions gives a concrete design principle that can be examined independently of headline error reductions.

  2. The insight is translated into identifiable operations. Equation (2) separates interaction generation from spectral weighting, while band- and type-dependent kernels express the proposed scale sensitivity. The shared factorization makes this organization feasible without a separate dense channel transform for every band. The construction uses established mathematics productively; its value lies in the resulting inductive bias rather than in claiming a new decomposition theorem.

  3. The findings suggest a useful boundary for the idea. Gains on Navier–Stokes and five NL6 tasks, together with weaker results on smoothing and interpolated-geometry settings, are consistent with a bias aimed at preserving and transforming active spectral interactions. The shared-band ablation and spectral analyses provide some mechanism-level support, although they do not fully explain the total advantage.

Critical weaknesses
  1. The operative insight is not yet isolated from generic multiplicative expressiveness. Bony's decomposition reorganizes a product; it becomes a distinct learned inductive bias through the separate treatment of scales and interaction types. In Table A2, a shared-band version remains relatively close to the full model, suggesting that much of the improvement may come from bilinear mixing, the two-branch design, or bandwidth choices. This does not make the classical decomposition uninteresting, but it leaves unclear which part of its structure is essential to the claimed advance over other nonlinear spectral operators.

  2. “Nonlinear PDE” is too broad a criterion for predicting when the method should help. The manuscript itself distinguishes smoothing maps from tasks with sustained high-frequency interactions, and NL6 mixes state evolution with parameter-to-state mappings. The practically useful insight may therefore concern the spectral structure of the learned map rather than the algebraic class of the governing PDE. Moreover, learned latent operands are not constrained to represent physical fields or derivatives, so the physical interpretation motivates the architecture without guaranteeing that its branches recover those physical mechanisms.

  3. The experimental protocol weakens verification of the mechanism and its practical advantage. Section 5 selects the lowest test error during training, which uses the test set for checkpoint selection. The reported smaller ablation differences need an independent evaluation protocol. Table A3 also reports 50.7 training hours for ParaNO versus 18.3 for HO-FNO, so fewer parameters do not establish better accuracy at matched computational cost. These issues qualify the evidence for the insight rather than replacing the conceptual assessment with a demand for more benchmarks.

Questions for authors
  1. What property of the Bony organization is essential to ParaNO's advantage beyond a generic bilinear spectral branch? Please interpret the relatively small full-versus-shared-band differences and clarify what the current ablations can attribute to scale/type structure.

  2. What property of an input–output map would let a practitioner anticipate an advantage before training? Is active cross-scale spectral interaction a more accurate criterion than PDE nonlinearity, and how should the latent-operand interpretation be understood?

  3. Do the main rankings and mechanism-level conclusions survive validation-selected checkpoints and a comparison at similar training cost? Please use these checks to delimit the evidence for the proposed bias, rather than treating parameter count as a substitute for compute.

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer anUY

We thank the reviewer for their constructive and interesting comments. We believe the points raised have helped sharpen the paper, and our detailed responses are provided below.

W1/Q1) Scale structure beyond bilinear mixing We agree with the reviewer that ParaNO's performance may result from several complementary architectural choices. However, we believe that Table A2 and Fig. A2 provide evidence for the specific contribution of band-dependent parametrisation. Under the full training protocol, with three seeds and a matched 2M parameter budget, band-dependent kernels consistently outperform shared kernels across four NL6 benchmarks, reducing error by 8% on average. Moreover, Fig. A2 shows that the learned kernels are genuinely different across bands (average similarities below 0.68), while remaining more correlated than random kernels. Adjacent bands also exhibit greater similarity in 36 out of 39 terms, suggesting that the model learns a structured dependence on frequency scales. We nevertheless agree that other architectural choices, particularly the separation into linear and nonlinear branches, may contribute substantially to performance. This separation is also mathematically motivated by the analysis of quasilinear PDEs. We plan to investigate these contributions as future work and thank the reviewer for highlighting this important distinction.

W2/Q2) Applicability and latent operands We agree that “nonlinear PDE” alone is too broad a criterion for predicting when ParaNO should help. Our motivating hypothesis concerns the spectral structure of the target input–output operator in the data regime considered: ParaNO is designed to model interactions between frequency scales that remain relevant to the prediction task. The algebraic and differential structure of the PDE helps motivate these interactions, but does not by itself determine their importance in the learned map. This distinction also clarifies the scope of NL6. Although its tasks involve different mapping types, their intended common feature is the need to resolve non-trivial spectral interactions. NL6 should therefore be interpreted as testing this architectural hypothesis across heterogeneous operators, rather than isolating PDE nonlinearity as a single explanatory factor.

Regarding the learned operands, we agree that they are not constrained to coincide with physical fields or derivatives. What the architecture enforces is an organisation of interactions between latent fields into low–high, high–low, and comparable-scale contributions. The channel projections in Eq. (4) do not mix spatial Fourier frequencies and commute with the dyadic filters, so learning the operands retains this scale organisation. We use this analogy only to motivate the preservation of scale structure, not to claim that the learned operands themselves represent derivatives or other identifiable physical quantities. ParaNO can therefore learn which latent features should interact while retaining the dyadic organisation required by the paradifferential inductive bias. Consistently, our band-dependence ablations support the usefulness of scale-dependent parametrisation in the tested tasks, without implying that individual branches recover specific physical mechanisms.

W3/Q3) Evaluation protocol and training cost Following the reviewers' suggestion, on Geo-PDE we now report test metrics at the final training epoch rather than the lowest test errors. Additionally, we redefine the data splits in The NL6 to have train, test, and validation sets. On The NL6 we perform checkpoint selection on the validation set and report test metrics with the selected checkpoint. We specify this evaluation protocol in the revised paper. The updated evaluation leaves ParaNO’s ranking unchanged; while some relative margins shift under the revised protocol, ParaNO retains substantial leads on all tasks where it ranks first. We also clarify that the ablations in Sec. A.3 already follow a separate fixed protocol: all variants are trained for the same reduced budget of 150 epochs and evaluated at the final epoch, without test-based checkpoint selection. Finally, our comparisons use matched parameter budgets and a common maximum number of training epochs; they are not intended as matched-wall-time comparisons. Indeed, Table A3 explicitly reports ParaNO’s higher training cost. Profiling shows that ParaNO’s runtime is dominated by explicitly creating the scale-separated interactions introduced by the architecture (51% of the training step). We report optimisation of this operation as future work.

Dear Reviewer anUY,

Thank you again for your thoughtful and constructive review.

As the discussion period is approaching its end, we would greatly appreciate hearing your thoughts on our rebuttal and the revised manuscript.

In particular, we have addressed the evaluation concerns through a corrected validation-based protocol, confirmed that our main conclusions remain unchanged, and provided additional evidence for the contribution of band-dependent parametrisation.

We have also substantially clarified the scope and interpretation of our architectural inductive bias, following your suggestions.

We kindly ask the reviewer to reconsider their judgement under these material, and evaluate our revision for its scientific merits. We remain available throughout the little remaining time for discussions."

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all the reviewers for their constructive feedback. We appreciate that reviewers found ParaNO clearly presented (anUY, xjbw, 1WKr), mathematically well motivated (anUY, xjbw, 1WKr), novel (xjbw), parameter-efficient (1WKr), and empirically promising (anUY, 1WKr, mxbF). We fully acknowledge the reviewers’ concerns regarding the evaluation protocol, positioning and scope, attribution of gains, and computational trade-offs. In the individual responses and revised manuscript we have provided robust evidence and explanations, addressing all the major issues raised thoroughly. In light of these additional clarifications and data provided, we respectfully request the reviewers to re-evaluate their initial assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.