acceptodds
Under review as a conference paper at ICLR 2027

Zero-shot generalization of transformer neural operators to larger domains

Abstract

Transformer-based neural operators have shown remarkable performance for approximating solution operators of partial differential equations on complex geometries. However, existing approaches implicitly assume a fixed domain size, which limits their ability to generalize at inference. In this work, we investigate domain extension, namely zero-shot inference on spatial domains that are significantly larger than those encountered during training. We argue that this setting fundamentally requires spatial locality and translation equivariance. We propose to implement this locality via a decomposable bias in the attention logits computation, enabling finely controllable locality while remaining fully decomposable into query–key inner products and directly compatible with optimized attention kernels. Combined with rotary positional embeddings, it enables expressive embeddings with controllable spatial support without altering the transformer architecture. We empirically show that our approach substantially improves zero-shot generalization to larger domains across two PDE benchmarks and a 3D industrial atmospheric flow application.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, transformer, generalization, atmospheric boundary layer, positional embedding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bv8XOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper attempts to design a transformer-based architecture for learning solution operators to partial differential equations which allows trained models to generalize to problems defined on larger spatial domains. This is done by combining Rotary Positional Encoding with distance biasing in such a way that the support of interaction kernel defined by an attention layer is restricted to interactions that are seen during training; this ensures that the learned kernel is preserved when applied to operators defined on larger spatial domains. Numerical experiments demonstrate the improved performance of the proposed architecture on operator learning tasks compared to other benchmarks.

Critical strengths

Domain extension for neural operators is an important problem in scientific machine learning. This paper provides a principled architectural design for this problem, with numerical experiments providing a proof of concept.

Critical weaknesses

The numerical experiments are quite limited, results are only provided for the shallow water wave equations, Gray-Scott equations, and microscale atmospheric time flow. Although the third problem is quite challenging, the limited experiments make it difficult to determine whether the proposed architecture will succeed in more general settings. In addition, the results are only compared to transformers with RoPE and L2L^2 bias. A more thorough numerical study and comparison with existing operator learning architectures is necessary to demonstrate the utility of the model.

Additionally, the architectural novelty seems limited, as it essentially combines two standard modifications to the transformer architecture in a clever way.

Questions for authors
  • Minor comment: I found it difficult to understand the motivation for the proposed architecture when reading the introduction. In Section 2, the motivation is explained as "preserving the same kernel when changing the domain size at inference time" -- I like this explanation, but it would be helpful to emphasis it earlier in the paper. In addition, the introduction mentions that non-locality is necessary for neural operators to be universal, but then goes on to talk about ways to enforce locality; it should be emphasized that the locality being enforced is for the kernel interaction, not the full operator.

  • As you mention at the end, restricting the interactions of the kernel to the training spatial domain could be limiting, since the true solution operator on a larger spatial domain may depend on the size of that domain. Yet, in your experiments, restricting the kernel interactions does not seem to hurt the performance of the model. Do you have a sense of why this is, and whether this depends on the structure of the equations you chose to test on?

  • In Section 2.3, you mention that the Gaussian bandwith λ\lambda should be "at most a third of the training domain size." By size, I'm assuming you mean the diameter of the training domain? And is it possible to test numerically the dependence on λ\lambda?

3 comments

Dear Reviewer bv8X,

Thank you for your time and consideration in reviewing our manuscript. Here are answers to your concerns and questions.

  • Novelty and numerical experiments

    We would like to clarify the novelty of our contribution. To the best of our knowledge, this is the first work that systematically study the zero-shot generalisability to larger physical domain which we defined as domain extension. Some previous efforts have been given to extend the generalization domain based on completely different approaches. Graph-based methods are inherently local and can be used in this context, but are too resource-hungry to run on large meshes. Domain decomposition can also be used [2], but the proposed method requires explicit sub-domain definitions and data over those domains, strongly constraining the data generation.

    Regarding the novel domain extension problem introduced in this paper, there are few existing works with which we can directly compare our approach, as almost all existing PDE benchmarks (The Well, DrivAerML, CFDBench realPDEbench, ...) and models focus on single-domain predictions. Consequently, each experiment requires custom datasets, hence running a large number of experiments is resource-intensive.

    Furthermore, our approach operates at the position-embedding level and is therefore agnostic to the overall architecture, provided that attention is applied between spatial points. We are not proposing yet another neural operator architecture. We have tested the performance of the proposed bias structure with different neural operators: the generic stack of transformer blocks in Section 3.1 and the AB-SWIFT architecture in Section 3.2. To complement this, we are also adding a comparison with the original formulation of FlashBias as an additional baseline for the academic cases. Our results show that FlashBias errors are twice as large as any other model for Gray Scott equations and 30 times as large for Shallow Water equations. More crucially, FlashBias error increases with the inference domain size, which we explain by the model predicting freely the decay slopes, independently of the training domain size. This further justify our unique approach to couple RoPE embeddings and a fixed geometry bias term. The comparison can be found at https://osf.io/sjv9k/overview?view_only=5fe647f9c14d4531af776e87a144faf9.

    We are also adding a multiscale graph baseline in the atmospheric case. We find lower performances for scales 1 and 2, while the model runs out of VRAM and can't infer on the scale 5. This further demonstrates the strength of our transformer-based approach over possible alternatives. A comparative figure illustrating predictions at scale 2 can be found in the repo linked above.

  • Motivation for a local architecture

    We are rephrasing parts of the introduction to clarify our idea, notably by stating more clearly that our approach rely on constraining the integration kernel to be local in order to make it robust to a change in domain size as you suggested.

  • Behavior of the true solution operator

The shallow-water equations and Gray-Scott equations are temporal, local equations where only information within a neighborhood impacts the solution at each time step. Hence, it makes sense that a local operator performs very well. The atmospheric flow case is much trickier, as we are predicting directly the steady-state solution from the input geometry and flow parameters. Hence, information can have an impact at infinite distances, and the solution operator will evolve with domain size. This is reflected in the numerical results, as performance decreases with scaling in the microscale atmospheric flow case (Table 3), which we attribute directly to new large-scale behaviors not learned by the model (Line 365).

  • Lambda sensitivity

    "Domain size" here refers to some notion of diameter, but is not defined more explicitly as a precise definition depends on the considered domain shape and axis (for anisotropic cases). We are adding a new sensitivity study for λ\lambda in the Gray-Scott case. Results show that performance decreases for λ>400\lambda > 400 for a domain of size 10001000 aligning with our analysis of 1/31/3 (which is slightly conservative), and the corresponding figure can be found in the repo linked above.

We would like to emphasize again that our goal is not to propose yet another neural operator architecture. Rather, we study a previously underexplored extrapolation setting, domain extension, and show that preserving kernel support through locality is a simple architectural principle that can be applied to existing transformer neural operators. We believe this perspective is the main conceptual contribution of the paper.

[2] Huang, J., Zhang, K., Wu, Y., & Cheng, Z. (2026, April). Operator learning with domain decomposition for geometry generalization in PDE solving. In International Conference on Learning Representations (Vol. 2026, pp. 69121-69148).

Thank you for your reply, which helps to clarify the scope of your contributions. I do think that identifying an architecture-agnostic design principle which improves model performance under domain extension is a meaningful contribution. I have raised my score to reflect this. I still think that the current presentation needs substantial work and that additional experiments would make a stronger case for the method.

Dear Reviewer bv8X,

Thank you for the score increase. We have uploaded the revised version of our manuscript, which adds FlashBias as an additional baseline in Section 3.1, and a comparison with a multiscale graph model for the atmospheric flow case in Appendix D. We have also added a new test case of microscale atmospheric featuring a real urban area in subsection 3.2.2, further demonstrating the interest of our method or industrial use cases and providing an example of the true solution operator evolving with domain size. Furthermore, the sensitivity study on λ\lambda can now be found in Appendix C. Finally, we have also reformulated parts of the introduction to clarify our approach, in particular related to the locality vs non-locality discussion.

Reviewer_9fW9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors addressed an important limitation of the current trend in PDE surrogates. While computer vision often uses images of different sizes (not just resolution, but larger FOV) during testing, PDE surrogates are mostly trained on a fixed domain. The primary idea is to use translation-invariant kernels, meaning the kernels must be local. The author also designed positional embeddings. Decomposability was also a design motivation for speed. The author showed the effectiveness of the method empirically on 3 problems.

Critical strengths
  • The author addresses an underaddressed gap in the current PDE surrogate study.
  • The author creates a dataset for this particular problem. If released, even though small, it will help the community.
Critical weaknesses
  • The technique/architecture introduced is well known and well used. Windowed attention (such as the Swin Transformer) and its plethora of adaptations in neural operators (such as POSEIDON) already contain such local interactions. And there are faster implementations of windowed attention [1]. At the same time, graph-based models trivially satisfy the design. These models should be evaluated and compared. Or the authors should explicitly explain why those models cannot be used for the experiments in the paper.
  • While scalability is a main concern, I do not find the training data resolution used in the experiments.
  • In [Line 053], the universal operator needs to be non-local, yet the authors are designing local models. This requires additional clarification
  1. Flash Window Attention: speedup the attention computation for Swin Transformer
Questions for authors
  • [line 048] "practical importance": What is a practical example?
  • [line 202] What are a and u in the flash bias?
  • Experiment: Is the resolution kept fixed when the domain is enlarged?
4 comments

Dear Reviewer 9fW9,

Thank you for your time and consideration in reviewing our manuscript. We confirm that datasets will be released publicly with the paper. Here are answers to your concerns and questions.

  • Novelty compared to existing architectures

    While window attention and Swin Transformers use local attention or local patching operations, they are only defined for structured grids (i.e. pixels/images). A key novelty of our work is making sliding-window-type locality possible for unstructured data, as it relies only on coordinates and unstructured point-cloud formulations. These are essential for industrial-scale problems with complex geometries, which rely on unstructured meshes.

    On the other hand, graph-based methods may be inherently local and compatible with unstructured data, but they scale poorly with mesh size in terms of computational resources. Additionally, they tend to perform worse than transformer-based models, especially over complex geometries and large meshes [1]. In particular, the scale-5 cases in microscale atmospheric flow contain about 22-33 million mesh cells, which is barely attainable with graph-based methods, even with significant work to partition the graph in order to reduce memory consumption. Simulations on real cases are completely beyond the range of graph methods, with meshes of the order of several tens of millions of cells. This is mentioned briefly in our state-of-the-art section (Line 426), which we will expand to better justify our approach. To illustrate this, we are adding a comparison with a multi-scale graph model [2] in the atmospheric case. Results show that its errors are about twice as high on the scale 1 and 2 cases, the model takes significantly longer to train and infer, and, more importantly, it doesn't have enough VRAM to run the scale 5 case on a single GPU. By contrast, transformer-based models have been scaled to hundreds of millions of mesh cells [3]. A comparative figure illustration predictions at scale 22 can be found at https://osf.io/hjtky/overview?view_only=c037475a32a44e44a641d23478709625.

  • Non-locality vs locality clarification

    We argue that while non-locality is a theoretical requirement for universality, domain extension is fundamentally a finite-data extrapolation problem over the space of all neural operator. In this regime, models have no opportunity to learn meaningful spatial information beyond the maximum distance covered by the training data, any predictions it makes beyond this distance would essentially be hallucinations. Hence, it makes sense to make the operator local in accordance with this bound. We are rephrasing parts of the introduction to clarify this idea.

  • Practical importance

    We demonstrate practical examples through the numerical experiments carried out; more generally, domain extension can be applied to any physics problem driven by local interactions, and allows one to learn a large-scale operator from simulations on small domains that are cheap to generate.

  • uu and α\alpha definitions

    α\alpha is a linear layer predicting slopes from the latent-field value at the query location, i.e. u(c)u(c) in accordance with the notation of our paper. This follows the definition of the FlashBias paper and was checked against their public implementation. We will rewrite the legend of Table~1 to make the definition clearer.

  • Resolution

    The resolution, i.e. mesh refinement, is kept fixed across all experiments and defined in Appendix D along with other data-generation details. Consequently, the total number of points in each simulation scales with the domain size.

[1] Wu, H., Luo, H., Wang, H., Wang, J., & Long, M. (2024). Transolver: A fast transformer solver for pdes on general geometries. arXiv preprint arXiv:2402.02366.

[2] Cao, Y., Chai, M., Li, M., & Jiang, C. (2023, July). Efficient learning of mesh-based physical simulation with bi-stride multi-scale graph neural network. In International conference on machine learning (pp. 3541-3558). PMLR.

[3] Alkin, B., Bleeker, M., Kurle, R., Kronlachner, T., Sonnleitner, R., Dorfer, M., & Brandstetter, J. (2025). AB-UPT: Scaling neural CFD surrogates for high-fidelity automotive aerodynamics simulations via anchored-branched universal physics transformers. arXiv preprint arXiv:2502.09692.

Dear Reviewer 9fW9,

Since we have uploaded our response six days ago and the rebuttal period ends on Wednesday, we were wondering if you had time to read our answers. We look forward to your feedback and further suggestions.

We have also uploaded the revised version of the manuscript. In particular, we have revised the introduction to clarify the locality vs non-locality discussion, and the state of the art to better explain the advantage of our approach compared to graph-based models or transformers with patching / shifted window. We have added the comparison with a multiscale graph model in Appendix D, and clarifications on resolution in Appendix F.

Dear Reviewer 9fW9,

As the rebuttal ends tomorrow, we were wondering if you had time to read our answers to your initial concerns and questions. We would like to know soon if you have any follow up questions or need any additional clarifications, so that we may have enough time to address them before the end of the rebuttal.

Dear Area Chair,

We posted our response to Reviewer 9fW9 more than a week ago and uploaded a revised manuscript three days ago. We have followed up twice and are still awaiting feedback. With less than 48 hours remaining in the rebuttal period, would you be able to bring our response and revised manuscript to the reviewer 9fW9’s attention? We would welcome the opportunity to address any follow-up questions before the deadline. Thank you for your help.

Reviewer_ubkLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the zero-shot generalization of Transformer-based neural operators to larger spatial domains. The authors combine RoPE with a decomposable distance bias and propose a hyperbolic bias that supports anisotropy and directional asymmetry. Experiments cover the one-dimensional shallow water equations, the two-dimensional Gray–Scott reaction–diffusion equations, and three-dimensional steady-state urban wind flow. The results show that the proposed method substantially improves domain-extension performance compared with RoPE.

Critical strengths
  1. The work is clearly motivated and practically relevant. The paper addresses zero-shot generalization to larger physical domains using only simulation data from smaller domains for training. Experiments on multiple PDE tasks and three-dimensional urban wind flow demonstrate improvements in domain-extension performance, supporting the effectiveness of the method.

  2. The method is relatively simple and offers a degree of physical interpretability. The authors use positional biases to explicitly control the range of spatial interactions and accommodate anisotropy and directional asymmetry according to the geometry, linking the model design to spatial dependencies in physical problems. The bias also preserves decomposability into query–key inner products.

Critical weaknesses
  1. The independent contribution of the asymmetric design lacks direct validation. The paper does not compare symmetric and asymmetric hyperbolic biases while holding other settings fixed. Consequently, the current results do not sufficiently distinguish the performance gains attributable to locality, anisotropy, and asymmetry.

  2. The comparison with the L2 baseline remains insufficient. The authors attribute the degradation of the L2 baseline at larger domain scales to numerical errors from low-precision computation, but do not sufficiently demonstrate an advantage over a numerically stable L2 implementation.

  3. The advantage of the hyperbolic formulation over simpler alternatives remains unclear. A shifted Gaussian bias is also decomposable and can represent an interaction region offset from zero displacement by shifting its center, yet the paper lacks comparisons with such alternatives.

Questions for authors
  1. Could you compare symmetric and asymmetric hyperbolic biases while keeping all other settings fixed?

  2. At computationally feasible, representative domain scales, does the hyperbolic bias still offer a clear advantage over an L2 baseline computed in Float32 or using a numerically verified implementation?

  3. Have you considered comparing against a shifted Gaussian bias, which is also decomposable?

  4. Could you clarify the overhead introduced by the additional bias channels to help assess the practical scalability of inference on larger domains?

5 comments

Dear Reviewer ubkL,

Thank you for your time and consideration in reviewing our manuscript. Here are answers to your concerns and questions.

  • comparing symmetric and asymmetric biases

    We agree that properly evaluating the advantages of an asymmetric bias over a symmetric one is important. To address this concern, we are adding a new test case based on simulations over a real urban area, where both the building zone and the wake zone are expanded. We show that a symmetric bias leads to spurious wakes behind the buildings, in the area beyond what the model saw during training. An asymmetric bias allows us to restrict interactions behind the buildings independently of those ahead of them and therefore does not face the same issue. A figure of the results can be found at https://osf.io/bh4pg/overview?view_only=9d4bcffea4cf466ebfa26011982e704d (with the full prediction and a zoom on the spurious wakes behind the buildings) and will be added to the paper.

    Our original setup did not fully demonstrate the advantage of the asymmetric bias, because only the building zone (which is symmetric) is expanded, while the wake zone (which is asymmetric) remains unchanged. As a result, the original experiment is not well suited to isolate the specific contribution of asymmetry. The additional urban test case addresses this limitation by providing a setting where the respective benefits of symmetric and asymmetric biases can be assessed directly.

  • Differences in L2 and hyperbolic

    Compared with the L2 bias proposed in FlashBias, the hyperbolic formulation's advantage is to offer a numerically stable decomposition for bfloat16 precision, and with less terms than the L2 formulation (in addition to the possibility to introduce asymmetry as discussed previously). Indeed, we show that both are mathematically equivalent for small displacements (Line 237 and Appendix A.2), hence no differences are expected in ranges where the L2 decomposition unstabilities don't manifest. This matches our experimental findings: both approach yield the same results in Table 2 for scalings up to 33.

    However, there are strong incentives to using Bfloat16 precision over Float32. It significantly reduces memory consumption and training and inference time. A quick test showed a ×2\times 2 VRAM cost a ×10\times 10 inference time between Bfloat16 and Float32 precision for Gray-Scott at scale 10 with RoPE. Additionally, most memory-efficient kernel implementations do not support Float32 precision (Notably all FlashAttention versions), causing Pytorch to fall back on non-efficient versions with quadratic memory cost when Float32 is used.

  • Shifted gaussian bias

    A shifted mean of the bias is an interesting idea. We note that it can be implemented for both the L2 and the hyperbolic formulation. Therefore, it should be considered an independent component. However, while the shifted mean indirectly results in asymmetry, it behaves differently and does not fully replace independently chosen, non-symmetric decay slopes of the hyperbolic formulation. Instead, a shifted mean induces a prior where the maximum interaction occurs for c−ξ=μc-\xi=\mu, while physics phenomena tend to decay with distance and hence have maximum interactions for c=ξc=\xi. Furthermore, while the hyperbolic formulation is more complex mathematically, it is decomposable in fewer terms (22 per spatial dimensions and 33 if a shift is added) than the L2 formulation (33 per spatial dim, and 55 with a shift) in addition to being more stable numerically.

  • Computational overhead

    We measure roughly a 40%40\% longer forward pass time with a bias than without during the forward pass (at scale 1). We note, however, that our implementation has not been optimized for speed and this number can be significantly reduced. In particular, machine learning frameworks are optimized for dimensions that are powers of two, but concatenating a bias to the query and keys results in latent dimensions that are no longer powers of two, resulting in a disproportionate computational cost relative to the number of added features. If inference speed is a concern, a more optimal implementation would be to replace some RoPE features instead of concatenating them, resulting in no overhead (beyond a single computation of the bias values) relative to standard attention, at the cost of a small number of available RoPE frequencies (22-33 per spatial dimension out of 6464) and hence expressivity. We did not implement this approach because inference speed was not a concern in this proof-of-concept work, and we preferred a fair comparison in terms of accuracy for each method, i.e. a constant hidden size for the RoPE part.

Dear Reviewer ubkL,

Since we have uploaded our response six days ago and the rebuttal period ends on Wednesday, we were wondering if you had time to read our answers. We look forward to your feedback and further suggestions.

We have also uploaded the revised version of the manuscript. In particular, the comparison with a real urban area demonstrating the utility of asymmetric biases can be found in Subsection 3.2.2.

Dear Authors,

Thank you for your detailed responses and the additional experiments. I appreciate your efforts to address my concerns, particularly the new urban test case comparing symmetric and asymmetric biases.

While the additional results and clarifications have addressed some of my concerns, I believe the advantages of the hyperbolic formulation over a numerically stable L2 implementation and simpler alternatives such as a shifted Gaussian bias still require further empirical validation.

Overall, I appreciate the authors' efforts to improve the manuscript. However, considering the remaining concerns, I have decided to maintain my original rating.

Thank you again for your thoughtful responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.