- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new network architecture in the field of Neural Operators, which learn the dynamics of PDEs. The architecture is designed to capture the non-linear responses that appear in the dynamics. Based on the idea that linear operations do not mix frequency components, the authors add a branch to FNO that explicitly mixes frequency components, so that the model can represent strong non-linearity. This branch is based on the Bony decomposition, which is also used in the analysis of non-linear PDEs, so a good fit can be expected.
Strictly speaking, FNO also learns interactions between frequency components implicitly. Still, making this explicit is a meaningful contribution.
Critical strengths
The performance also seems consistently high. However, this is hard to evaluate until the problem below (the test data is used for model selection) is resolved.
Critical weaknesses
(1) The major problem is the handling of the results. The following sentence suggests that the reported performance is chosen by looking at the test data, which is an improper use of the data. At minimum, a fair selection procedure is needed, such as a train–valid–test split, or stopping at a fixed number of iterations matched to prior work without looking at the test set.
- l310 "we evaluate and compare models in terms of the lowest relative ℓ2 test errors achieved during training"
(2) The motivation is unclear.
- The abstract says "capable of rigorously resolving complex non-linear dynamics", but the evaluated problems actually include linear dynamics. It is natural that even linear dynamics produce an output that is non-linear with respect to the input, and calling this "non-linear" is a misuse of the term.
In addition, the paper appears to contain many small errors in its descriptions. Below I list at least the ones that are clearly contradictory. Given this many errors, there are probably more that cannot be predicted from the manuscript, as well as ones I have overlooked. At minimum, these must be thoroughly corrected.
(3) In the first half of the paper, the authors acknowledge that FNO performs frequency mixing, but in the second half they deny it. It is true that FNO is less explicit than the proposed method, but this is somewhat of an overclaim.
- l138 "While this implicit coupling is theoretically universal, it is structurally agnostic to the underlying PDE and the true spectral interactions."
- l442 "Rather than ignoring cross-frequency mixing (like FNO) or mixing modes uniformly (like HO-FNO), ParaNO groups frequencies into interacting octaves and routes them directly through the distinct algebraic regimes depicted in Fig. 3:"
(4) This work introduces interactions between frequency components. That is fine, but I question calling it an "inductive bias," as in the sentence below. This term is usually used when the learning procedure or the architecture is given a property such as equivariance, or when some implicit regularization arises. A new computational term by itself is generally not called an inductive bias. At minimum, the authors should explain its effect in relation to the properties of the target being learned (for example, that it is strongly related to some specific property of PDEs).
- l144 "we introduce an inductive bias that explicitly models the frequency interactions and resonant behaviours that characterise the vast majority of governing PDEs."
(5) Also, FNO does not apply a non-linear activation inside its spectral branch either (the activation is applied after the branch), so the following sentence is factually incorrect.
- l244 "Crucially, unlike FNO, we omit any non-linear activation within this branch,"
(6)
- Table 1: "all baselines retain their original codebase"
- l317: "As CATO has not publicly released an official implementation at the time of writing,"
(7)
- Table 1: "While retrained,"
- l321: "The sole exception is Plasticity, as we were unable to reproduce the baseline metrics despite extensive efforts, we report the originally published values for this task."
(8)
- l338: "Across Geo-PDE (Tab. 1), ParaNO excels on Native Grids containing active spectral mixing, improving upon HO-FNO by 23% on Navier–Stokes. However, it underperforms relative to spatial methods (like CATO and MSPT) on Darcy flow and most interpolated geometries."
In fact, according to Table 1, ParaNO is inferior to MSPT only on PIPE.
(9)
- Abstract: "reduces errors by factors of 2× to 4.6× on the NL6"
In fact, ParaNO is inferior on HYP, and the reduction on VAW is 1.6×.
(10)
- l53: "we also introduce the Non Linear Six (the NL6) benchmark, a dataset of six non-linear PDEs that exhibit these dynamics, numerically simulated on periodic grids."
- l278: "While VAW and DNO simulate state evolutions that are technically linear, they represent highly non-linear mappings with respect to their heterogeneous input fields, testing a model's ability to process variable fractional orders and analytic non-polynomials, respectively." If so, they are linear PDEs.
Questions for authors
What is the performance when it does not depend on the test data? If it is comparable to the current manuscript, that would be a strong reason for acceptance.
4 comments
Dear Reviewer mxbF,
Thank you again for your time and thoughtful review. As the discussion closes soon, November 18 (AoE), we would appreciate hearing whether our rebuttal addresses your concerns. We would be happy to clarify any remaining points before the deadline.
All the Best, The authors
I thank the authors for the substantial revision and response.
(1) I am glad that the authors acknowledged the problem in the original evaluation protocol. However, Tables A3 and A4 still appear to contain the same problematic results as those in the first submission. The same probably holds for the other figures and tables that were not replaced (e.g., Fig. 1)
I cannot agree with the response to (4). The term "architectural inductive bias" sounds even more like CNNs or GNNs. I think a description such as "an architecture inspired by paradifferential calculus" would be reasonable.
The response to (5) appears inconsistent with Eq. (6). I cannot determine what the actual structure is.
In addition, the substantial revision seems to introduce various inconsistencies. For example, Table A5 summarizes seven benchmarks, while its caption says "The five standard benchmark problems."
It was not easy to find the time to read the entire manuscript very carefully within the limited discussion phase, but many errors still seem to remain. Although some of the issues have been fixed, the manuscript does not appear to be ready for publication. Therefore, it is difficult for me to change my rating.
We thank the reviewer for taking the time to go through our revised manusript, we appreciate that time in the discussion phase is limited.
-
We thank the reviewer for noticing this minor mismatch, we have now updated both tables. We would like to clarify that, for computational reasons, Table A3 was obtained by tuning the models on Navier Stokes and then testing them on The NL6. Models now adopt their last training checkpoint following the revised protocol. Note that this did not change the overall outcome of our experiment. Figure 1 has also been updated, but, as the reviewer may appreciate, changes are barely visible because metrics changed only minimally. We also confirm that all other figures and tables are correct and up to date.
-
We would like to clarify our use of the terminology. The concept of inductive bias is considerably broader than equivariance or invariance, which are a particular case of it. Mitchell (1980) defines inductive bias in terms of the assumptions or preferences that guide a learning algorithm towards particular generalizations, which is precisely our case. More specifically, Battaglia et al. (2018, Box 2) explicitly identify architectural structure as a source of inductive bias.
In ParaNO, the bias does not arise from merely introducing a bilinear operation, nor from Bony's decomposition alone, which is an algebraic identity. Rather, it arises from the structured parametrisation of bilinear spectral interactions: the architecture explicitly distinguishes low–high, high–low, and comparable-scale interactions, assigning learnable responses conditioned on both interaction type and frequency band, while sharing channel-mixing parameters across bands.
This introduces a preference for scale-structured bilinear operators rather than treating all frequency interactions uniformly. We therefore believe that architectural inductive bias is an appropriate and established description of this design principle, while an architecture inspired by paradifferential calculus accurately describes its mathematical motivation. The two descriptions are complementary rather than mutually exclusive.
-
Eq. 6 is correct and consistent with response number 5. Eq. 6 defines the whole ParaNO layer as:
Here, the activation function is applied to the linear combination of the linear () and non-linear () branches. As it can be observed from Eq. 2 of the FNO paper, the activation function is applied to what they define as , with being defined at their Eq. 4. This is equivalent to our linear branch . Their Eq. 2 thus corresponds to ). is therefore correctly classified as "the pre-activation transform of a FNO layer" and it includes all the FNO transformations except the activation function.
-
Once again, we thank the reviewer for noticing this minor caption inconsistency introduced while updating the table to include two additional rows. We appreciate that even minor inconsistencies like this contribute to the overall quality of the paper, but we are also happy to see that outstanding issues appear related mostly to very minor details and not to fundamental issues with our method or new experimental protocol.
We genuinely hope that our responses, who have now addressed all the reviewer's concerns, can motivate a score increase in alignment with their initially declared intention.
References
- Mitchell, T. M. (1980). The Need for Biases in Learning Generalizations.
- Battaglia, P. W., et al. (2018). Relational Inductive Biases, Deep Learning, and Graph Networks.
Official Response to Reviewer mxbF
We thank the reviewer for their time and efforts invested in our paper. Our detailed responses are given below. As the revised evaluation shows that performance remains comparable when selection is independent of the test data, we hope this directly addresses the reviewer’s stated criterion that such a result would provide "a strong reason for acceptance".
W1) Test-set selection and updated results We agree with the reviewer that our experimental procedure had to be improved. Therefore, we implement both the reviewer's suggestions. on Geo-PDE we now report test metrics at the final training epoch rather than the lowest test errors, and we specify this evaluation protocol in the revised paper. Additionally, we redefine the data splits in The NL6 to have train, test, and validation sets. On The NL6 we perform checkpoint selection on the validation set and report test metrics with the selected checkpoint. The updated evaluation leaves ParaNO’s ranking unchanged; while some relative margins shift under the revised protocol, ParaNO retains substantial leads on all tasks where it ranks first.
W2) Nonlinear dynamics versus interactions We thank the reviewer for pointing out this nuanced distinction. We agree that nonlinear PDE dynamics and nonlinear solution operators are not equivalent concepts. We have replaced "non-linear dynamics" with "non-linear interactions" in the abstract and text. The updated term accurately reflects the multiplicative coupling between fields and spatially varying coefficients that Bony's decomposition is designed to resolve. This clarification does not change our architectural motivation, which is to introduce a paradifferential inductive bias for learning operators governed by cross-scale frequency interactions.
W3) Implicit versus explicit frequency mixing We intended to distinguish implicit activation-driven mixing from explicit scale-structured interactions, and have revised the wording accordingly.
W4) Architectural inductive bias We use "inductive bias" in its standard architectural sense: a structural assumption that biases the learner toward some patterns or representations over others. ParaNO does not merely add a computational term, but imposes Bony’s low–high, high–low, and remainder interactions learned by the network. This mathematically-grounded structure directly reflects the scale organisation of non-linear products in paradifferential calculus, and we now make this explicit by referring to it as an "architectural" inductive bias.
W5) Activation placement in FNO We thank the reviewer for noticing this small phrasing ambiguity. We have clarified that Sθ corresponds to "the pre-activation transform of a FNO layer" and stressed that, "unlike a complete FNO layer, Sθ does not include the subsequent non-linear activation".
W6/W7) Incomplete Table caption We apologise for the omission in the Table's caption, we have now improved its alignment with the main text.
W8) MSPT comparison We have corrected this comparison by removing MSPT from the sentence.
W9) NL6 improvement range We thank the reviewer for noticing this minor discrepancy, which now specifies the improved performances in the correct ranges (which have been updated in accordance with the improved experimental protocol) and limits the claim to five of the six PDEs.
W10) PDE linearity versus task nonlinearity We agree that the terminology was imprecise. We have therefore revised the manuscript to refer to NL6 as six PDE-related learning tasks exhibiting non-linear interactions, broadening the terminology to encompass non-linearities arising in the target input–output map rather than only in the governing PDE.