- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper attempts to design a transformer-based architecture for learning solution operators to partial differential equations which allows trained models to generalize to problems defined on larger spatial domains. This is done by combining Rotary Positional Encoding with distance biasing in such a way that the support of interaction kernel defined by an attention layer is restricted to interactions that are seen during training; this ensures that the learned kernel is preserved when applied to operators defined on larger spatial domains. Numerical experiments demonstrate the improved performance of the proposed architecture on operator learning tasks compared to other benchmarks.
Critical strengths
Domain extension for neural operators is an important problem in scientific machine learning. This paper provides a principled architectural design for this problem, with numerical experiments providing a proof of concept.
Critical weaknesses
The numerical experiments are quite limited, results are only provided for the shallow water wave equations, Gray-Scott equations, and microscale atmospheric time flow. Although the third problem is quite challenging, the limited experiments make it difficult to determine whether the proposed architecture will succeed in more general settings. In addition, the results are only compared to transformers with RoPE and bias. A more thorough numerical study and comparison with existing operator learning architectures is necessary to demonstrate the utility of the model.
Additionally, the architectural novelty seems limited, as it essentially combines two standard modifications to the transformer architecture in a clever way.
Questions for authors
-
Minor comment: I found it difficult to understand the motivation for the proposed architecture when reading the introduction. In Section 2, the motivation is explained as "preserving the same kernel when changing the domain size at inference time" -- I like this explanation, but it would be helpful to emphasis it earlier in the paper. In addition, the introduction mentions that non-locality is necessary for neural operators to be universal, but then goes on to talk about ways to enforce locality; it should be emphasized that the locality being enforced is for the kernel interaction, not the full operator.
-
As you mention at the end, restricting the interactions of the kernel to the training spatial domain could be limiting, since the true solution operator on a larger spatial domain may depend on the size of that domain. Yet, in your experiments, restricting the kernel interactions does not seem to hurt the performance of the model. Do you have a sense of why this is, and whether this depends on the structure of the equations you chose to test on?
-
In Section 2.3, you mention that the Gaussian bandwith should be "at most a third of the training domain size." By size, I'm assuming you mean the diameter of the training domain? And is it possible to test numerically the dependence on ?
3 comments
Thank you for your reply, which helps to clarify the scope of your contributions. I do think that identifying an architecture-agnostic design principle which improves model performance under domain extension is a meaningful contribution. I have raised my score to reflect this. I still think that the current presentation needs substantial work and that additional experiments would make a stronger case for the method.
Dear Reviewer bv8X,
Thank you for the score increase. We have uploaded the revised version of our manuscript, which adds FlashBias as an additional baseline in Section 3.1, and a comparison with a multiscale graph model for the atmospheric flow case in Appendix D. We have also added a new test case of microscale atmospheric featuring a real urban area in subsection 3.2.2, further demonstrating the interest of our method or industrial use cases and providing an example of the true solution operator evolving with domain size. Furthermore, the sensitivity study on can now be found in Appendix C. Finally, we have also reformulated parts of the introduction to clarify our approach, in particular related to the locality vs non-locality discussion.
Dear Reviewer bv8X,
Thank you for your time and consideration in reviewing our manuscript. Here are answers to your concerns and questions.
Novelty and numerical experiments
We would like to clarify the novelty of our contribution. To the best of our knowledge, this is the first work that systematically study the zero-shot generalisability to larger physical domain which we defined as domain extension. Some previous efforts have been given to extend the generalization domain based on completely different approaches. Graph-based methods are inherently local and can be used in this context, but are too resource-hungry to run on large meshes. Domain decomposition can also be used [2], but the proposed method requires explicit sub-domain definitions and data over those domains, strongly constraining the data generation.
Regarding the novel domain extension problem introduced in this paper, there are few existing works with which we can directly compare our approach, as almost all existing PDE benchmarks (The Well, DrivAerML, CFDBench realPDEbench, ...) and models focus on single-domain predictions. Consequently, each experiment requires custom datasets, hence running a large number of experiments is resource-intensive.
Furthermore, our approach operates at the position-embedding level and is therefore agnostic to the overall architecture, provided that attention is applied between spatial points. We are not proposing yet another neural operator architecture. We have tested the performance of the proposed bias structure with different neural operators: the generic stack of transformer blocks in Section 3.1 and the AB-SWIFT architecture in Section 3.2. To complement this, we are also adding a comparison with the original formulation of FlashBias as an additional baseline for the academic cases. Our results show that FlashBias errors are twice as large as any other model for Gray Scott equations and 30 times as large for Shallow Water equations. More crucially, FlashBias error increases with the inference domain size, which we explain by the model predicting freely the decay slopes, independently of the training domain size. This further justify our unique approach to couple RoPE embeddings and a fixed geometry bias term. The comparison can be found at https://osf.io/sjv9k/overview?view_only=5fe647f9c14d4531af776e87a144faf9.
We are also adding a multiscale graph baseline in the atmospheric case. We find lower performances for scales 1 and 2, while the model runs out of VRAM and can't infer on the scale 5. This further demonstrates the strength of our transformer-based approach over possible alternatives. A comparative figure illustrating predictions at scale 2 can be found in the repo linked above.
Motivation for a local architecture
We are rephrasing parts of the introduction to clarify our idea, notably by stating more clearly that our approach rely on constraining the integration kernel to be local in order to make it robust to a change in domain size as you suggested.
Behavior of the true solution operator
The shallow-water equations and Gray-Scott equations are temporal, local equations where only information within a neighborhood impacts the solution at each time step. Hence, it makes sense that a local operator performs very well. The atmospheric flow case is much trickier, as we are predicting directly the steady-state solution from the input geometry and flow parameters. Hence, information can have an impact at infinite distances, and the solution operator will evolve with domain size. This is reflected in the numerical results, as performance decreases with scaling in the microscale atmospheric flow case (Table 3), which we attribute directly to new large-scale behaviors not learned by the model (Line 365).
Lambda sensitivity
"Domain size" here refers to some notion of diameter, but is not defined more explicitly as a precise definition depends on the considered domain shape and axis (for anisotropic cases). We are adding a new sensitivity study for λ in the Gray-Scott case. Results show that performance decreases for λ>400 for a domain of size 1000 aligning with our analysis of 1/3 (which is slightly conservative), and the corresponding figure can be found in the repo linked above.
We would like to emphasize again that our goal is not to propose yet another neural operator architecture. Rather, we study a previously underexplored extrapolation setting, domain extension, and show that preserving kernel support through locality is a simple architectural principle that can be applied to existing transformer neural operators. We believe this perspective is the main conceptual contribution of the paper.
[2] Huang, J., Zhang, K., Wu, Y., & Cheng, Z. (2026, April). Operator learning with domain decomposition for geometry generalization in PDE solving. In International Conference on Learning Representations (Vol. 2026, pp. 69121-69148).