- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors present a method for constructing and training neural operators (NO) to improve generalisability across parameters and boundary conditions. This is done by using a PDE parameter-dependent kernel in the NO, embeddings of BCs as DIrichlet BCs, and with a minimax-type loss. Several PDEs are then tested and cost accuracy is compared with standard numerical schemes.
Critical strengths
- The paper addresses an important problem in surrogate modelling for PDEs, namely generalisability and robustness to changes in PDE parameter and boundary conditions. This is an under-developed area of research in this field. The presented method seems to propose a simple yet sensible way of improving baselines meaningfully. Furthermore, they perform good comparisons with standard numerical methods.
Critical weaknesses
- The paper lacks clear mathematical exposition of certain aspects of the method which lead to ambiguity and may make reproducibility an issue. Maps which could be described more precisely are explained with words and references which make the method obscure.
Questions for authors
-
Could you use something other than angle brackets for concatenation? This is quite confusing.
-
How is the boundary condition map implemented precisely?
-
And could the same results be achieved simply by using a different network for each PDE parameter "bin"?
4 comments
W1 Explanation on Function Maps
We have revised Sections 4.2 and 4.3 so that every map in the model is defined by an explicit equation, with the changes highlighted in red. The encoder splits the input field into non-overlapping patches and embeds them with an MLP, and the decoder applies an MLP and reassembles the patches into the field. The PDE parameters are embedded by an MLP, and a separate MLP produces one score for each kernel. Each base kernel is a self-attention layer with its own query, key, and value weights, so the kernel in Equation 2 is given by the attention weights. The parameter and boundary tokens are added in front of the latent state tokens, serve as the context set in Equation 2, and are removed after each layer. The boundary transfer operator embeds the boundary data along the domain edges as boundary tokens, maps the boundary type to a learned embedding, and updates the boundary tokens with a self-attention layer over the boundary and interior tokens to form the latent Dirichlet representation. The gate selects the kernel with the highest score in the forward pass, and a straight-through estimator passes gradients through the softmax probabilities in the backward pass, so the partition of the parameter space is learned during training. Appendix S gives the full pseudocode, and we will release our code upon publication.
Thank you for your constructive feedback. We have updated the manuscript to incorporate your suggestions. We hope this clears up any confusion and concerns, and if so, we would greatly appreciate your consideration in raising the score.
I thank the authors for their comprehensive response and clarifications. These help the clarity of the paper and answer my concerns.
I have increased my score accordingly.
We sincerely thank the reviewer for the positive feedback. We greatly appreciate your support and your high evaluation of our work.
We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.
Q1 Concatenation Notation
We will denote concatenation by concat(⋅). The first term of Equation 2 then reads Wconcat(θ~,h(l),v(l)(x)). We thank the reviewer for suggesting this change and it will improve the readability of the equations.
Q2 Boundary Condition Map
The boundary encoder patchifies the boundary data along the domain edges, analogous to the patch embedding in ViT, and embeds each boundary patch as a boundary token. The boundary type (periodic, Dirichlet, or Neumann) is mapped to a learned type embedding. These are concatenated with the current latent state tokens, and a self-attention layer updates the boundary tokens using both the type embedding and the interior latent state. The updated boundary tokens form the latent Dirichlet representation. They are prepended to the latent state tokens together with a PDE parameter token produced by an MLP, and the sequence is processed by the kernel selected by the parameter gate. The boundary and parameter tokens are then removed, so only the latent state tokens pass to the next layer.
Q3 Separate Network per Parameter Bin
A separate network per parameter bin differs from our mixture of kernels in three ways. First, each layer has its own gate, so the number of kernel combinations grows exponentially (KL, where K is number of kernel and L is the number of layers) with depth while the number of kernels grows only linearly (K∗L). Covering the same combinations with separate networks would require significantly more networks, which could be infeasible to train and store.
Second, all components other than the attention kernels, including the encoder, decoder, MLPs, boundary transfer operator, and parameter embedding, are shared across all parameter values. These shared components learn the structure common to all parameter regimes so every parameter region benefits from the data of all others. A separate network per bin only sees the data in its own bin which could be insufficient to learn good representations and is also computationally expensive.
Third, the partition of the parameter space is learned from data and composed across layers. The resulting partition is not known in advance. Having separate networks would require choosing bins by hand and injecting human priors about where the physical regimes change, which are generally unknown. This would overly rely on human designers and could also limit the model’s expressiveness.
To definitely answer this question, we report a comparison with separate networks trained on 10 bins of the incompressible Navier–Stokes parameter range. The results show that the single pretrained model has much better performance than the single networks.