acceptodds
Under review as a conference paper at ICLR 2027

Scalable Operator Learning via Tensorized Transformer

Abstract

Attention-based models are widely used to learn solution operators for partial differential equations (PDEs), but the computational cost of attention grows quadratically with the number of tokens. Windowed attention reduces this cost but limits direct interactions to local neighborhoods. Factorized attention attends along subsets of coordinates, but its approximation properties and practical benefits remain unclear. We study a tensorized Transformer that stacks independently parameterized blocks, each attending along a single coordinate, so that one sweep over all coordinates yields global interaction. Viewing attention as an integral operator, we approximate kernels by finite sums of tensor products of one-dimensional kernels. We then prove that tensorized Transformers approximate continuous solution operators uniformly on compact sets. We compare against full and windowed attention under matched model and training configurations on four two- and three-dimensional benchmarks from The Well. Across three model sizes, tensorized attention reduces VRMSE by 17.5% relative to these alternatives and runs 2.1 faster than full attention.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific machine learning, operator learning, partial differential equation, kernel decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies tensorized Transformers for PDE operator learning. The model applies independently parameterized Transformer blocks along individual coordinates, allowing global interaction after a complete coordinate sweep. The paper provides a kernel-tensorization interpretation and a universal approximation result. Experiments compare full, windowed, and tensorized attention on four PDE benchmarks, including one three-dimensional benchmark, across three model sizes.

Critical strengths
  1. The controlled comparisons with full and windowed attention use the same encoder, decoder, model configurations, and training settings. The results show a useful accuracy–cost trade-off across several PDEs and model sizes, supported by measured inference time and ten-step rollout errors.

  2. The kernel-tensorization view provides a mathematical explanation for composing coordinate-wise attention. The approximation analysis establishes that this model class can approximate continuous solution operators under the stated assumptions.

Critical weaknesses
  1. The controlled comparison does not include representative linear-complexity or low-rank attention methods. Although the paper compares against FactFormer, A ViT, and MORPH, these retain their own architectures rather than serving as attention replacements within the same framework. It remains unclear whether tensorized attention offers a better accuracy–cost trade-off than alternatives such as kernel-based linear attention or Nyström attention. Adding one or two representative baselines with the same encoder, decoder, and training setup, and reporting complexity, runtime, peak memory, and prediction error, would strengthen the evaluation.

  2. The experiments provide limited evidence for why tensorized attention improves accuracy. The approximation theorem establishes expressivity with sufficient capacity, but does not explain the gains at the finite model sizes and training budgets used here. The head-count experiment in Figure 8 offers indirect support for the hypothesis that one-dimensional factors are easier to learn, but does not distinguish better optimization, reduced overfitting, or a more suitable spatial inductive bias. For windowed attention, the difference may instead come from access to global interactions after one sweep. Training and validation curves, together with targeted ablations of coordinate coupling or window size, would help clarify these explanations.

Questions for authors

1.How does tensorized attention compare with a representative kernel-based linear attention or Nyström approximation under the same framework and training setup, considering both prediction error and measured computational cost?

2.Does full attention have higher training error, or mainly a larger gap between training and validation error? Can controlled changes to coordinate coupling and window size help identify the source of the accuracy gains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer eLZP

Thank you for your helpful review. We sincerely appreciate your time and effort and have revised our paper based on your constructive suggestions. The revised content is temporarily appended to the end of the appendix, and we will incorporate the key content into the main text as space allows.


W1 and Q1: Linear and Nyström attention

Thank you for this insightful suggestion. We agree that adding linear-complexity and low-rank attention strengthens the comparisons. On RB, we use Performer [7] as a linear-complexity baseline and Nyström attention [8] as a low-rank baseline for full attention. Fig.11 (left, page 30) shows that our tensorized method achieves lower prediction error at comparable computational costs.

Inspired by the reviewer’s suggestion, we further expand the comparisons. Full, windowed, and tensorized attention are three different attention patterns, whereas softmax, linear, and Nyström attention are different attention implementations. Each implementation can be used with all three patterns. Fig.11 (right, page 30) compares the accuracy and computational cost of the three patterns implemented with Nyström attention. Tensorized attention consistently achieves lower prediction error than the other two patterns. The VRMSE gap between full and tensorized attention is larger than with softmax attention.

The latter results also support our motivation that kernel decomposition can simplify the features to be learned. With a low-rank approximation, full attention may struggle to represent a complex kernel, while tensorized and windowed attention can remain effective with simpler kernel structures.

We sincerely thank the reviewer for this suggestion, which also helps clarify why tensorized attention can be more effective than full attention.

  • [7] Choromanski et al. Rethinking Attention with Performers. ICLR, 2021.
  • [8] Xiong et al. Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention. AAAI, 2021.

W2: Limited evidence for why tensorized attention improves accuracy

Thank you for pointing out this limitation of our paper. We acknowledge that our original experiments provide limited evidence for the hypothesis that decomposed kernel factors are easier to learn. The low-rank attention comparisons suggested in W1 help clarify this motivation, as discussed above. We will use these experimental results to strengthen the discussion in the main text.

W2 and Q2: Training and validation curves

Thank you for this helpful suggestion. We have added training and validation curves for models on RB and MHD as two representative samples (Fig.12, page 30). Tensorized attention achieves lower final training MSE and validation VRMSE in both settings. This supports improved fitting under the same training budget, consistent with our hypothesis that decomposed kernel factors are easier to learn.

W1, W2, and Q2: Coordinate-coupling ablation

Thank you for reminding us of this important ablation. In our tensorized method, we sequentially couple coordinates by composing consecutive Transformer blocks, each attending along a different coordinate. This is a major difference from other factorizations like AViT and MORPH, which couple coordinates in parallel and combine the outputs by averaging or summation. We therefore ablate the way coordinates are coupled to (i) clarify the benefit of our coordinate coupling and (ii) strengthen the comparison with parallel factorizations.

The results in Fig.13 (right, page 30) show a consistent accuracy gain from our sequential coupling over parallel coupling. Experimental details are provided alongside the figure.

W2 and Q2: Window-size ablation

Thank you for reminding us of this important ablation. We ablate the window size of the large windowed attention models. Table 13 (page 29) summarizes the window-size ablation results.


Again, we thank the reviewer for the helpful comments and suggestions that have improved our paper. We appreciate the reviewer’s time and effort and would be grateful for any further thoughts on our responses.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors’ efforts, which have substantially improved the paper.

However, I still find the central motivation insufficiently supported. While the new experiments demonstrate empirical improvements, they do not convincingly explain why tensorized attention outperforms full attention. A deeper analysis of the attention patterns and underlying mechanisms is still needed.

Therefore, I maintain my original rating of 2 (Weak Rejection).

Reviewer_FKU4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the tensorized Transformer, a scalable operator learning architecture that decomposes multi-dimensional attention kernels into sequential compositions of 1D coordinate-wise attention blocks.

Its main contribution is a theoretical framework proving that kernel tensorization uniformly approximates continuous PDE solution operators on compact sets while reducing computational complexity from quadratic to quasi-linear relative to token count. The authors further demonstrate that composing independent coordinate blocks sequentially generates global cross-dimensional token interactions after a single sweep, outperforming existing parallel factorized attention models.

Validation across four two- and three-dimensional PDE benchmarks from The Well shows that the tensorized Transformer reduces VRMSE by 17.5% compared to full and windowed attention baselines. Additionally, it achieves a 2.1x speedup over full attention while maintaining superior scalability across multiple model sizes.

Critical strengths
  1. Rigorous Theoretical Grounding via Kernel Tensorization

The paper reframes attention as an integral operator and provides a formal proof (Theorem 4.1) showing that sequential compositions of 1D tensorized Transformer blocks with pointwise MLPs uniformly approximate continuous PDE solution operators on compact sets. Factorized and axial attention methods in deep learning are frequently treated as ad-hoc, heuristic speedup tricks. Establishing a rigorous theoretical foundation with universal approximation guarantees elevates tensorized attention to a principled mathematical paradigm for operator learning.

  1. Principled Structural Superiority of Sequential Block Composition

The authors demonstrate that stacking independent 1D blocks sequentially (Sd∘⋯∘S1\mathcal{S}_d \circ \dots \circ \mathcal{S}_1) interspersed with pointwise MLPs allows a single sweep to produce cross-product kernel terms (K2K1\mathcal{K}_2 \mathcal{K}_1) that achieve full global token interaction across all coordinates.

This resolves a key architectural limitation of prior factorized operator baselines (such as AVIT or MORPH), which use parallel sums or shared projections that fail to form multiplicative product kernels after one pass. It directly accounts for the empirical results: achieving a 17.5% reduction in VRMSE alongside a 2.1x speedup over full attention on complex 2D and 3D spatiotemporal benchmarks.

Critical weaknesses
  1. Rigid Dependency on Structured Cartesian Grids

The tensorized Transformer relies on performing sequential attention sweeps along orthogonal spatial axes (xx, yy, zz), which fundamentally requires data sampled on a regular Cartesian product grid.

  1. Lack of Zero-Shot Super-Resolution and Discretization Transfer Verification

A defining property of neural operators is discretization invariance—the ability to train a model on a low-resolution grid and evaluate it zero-shot on a higher-resolution grid. However, the paper evaluates models exclusively on fixed spatial grids matching the exact resolution used during training (e.g., 256×256256 \times 256 or 64364^3).

Questions for authors

N.A.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FKU4

Thank you for your thoughtful review. We sincerely appreciate your time and effort in reviewing our paper. We have carefully considered your suggestions and revised the paper accordingly.


W1. Rigid Dependency on Structured Cartesian Grids

Thank you for pointing out this major issue. We acknowledge that our tensorization method requires data to be sampled on a structured multidimensional grid and represented as a tensor. We have carefully noted this limitation in our conclusion. This limitation is shared by earlier factorized methods and state-of-the-art foundation models such as Walrus [3] and Poseidon [4].

For complex geometries, we agree that additional designs are needed to handle unstructured grids. One practical extension is to encode unstructured inputs onto a regular latent grid, following GINO [5] or adapting UPT’s supernode encoding [6], and apply our tensorized module there. We leave this extension to future work.

  • [3] McCabe et al. Walrus: A Cross-domain Foundation Model for Continuum Dynamics. ICML, 2026.
  • [4] Herde et al. Poseidon: Efficient Foundation Models for PDEs. NeurIPS, 2024.
  • [5] Li et al. Geometry-Informed Neural Operator for Large-Scale 3D PDEs. NeurIPS, 2023.
  • [6] Alkin et al. Universal Physics Transformers: A Framework for Efficiently Scaling Neural Operators. NeurIPS, 2024.

W2. Lack of Zero-Shot Super-Resolution and Discretization Transfer Verification

Thank you for this helpful suggestion. We agree on the importance of zero-shot super-resolution experiments. We trained three models using full, windowed, and tensorized attention on AM fields subsampled to 128×128 and evaluated them zero-shot on the original 256×256 fields. Table 12 (page 29) reports their one-step VRMSE, showing that tensorized attention achieves the best discretization transfer performance among the three attention patterns in this experiment.


We again thank the reviewer for the helpful comments, suggestions, and guidance for improving our paper. We sincerely appreciate the reviewer’s time and effort in reviewing our paper.

Thanks to the author for the additional verification on zero-shot super-resolution abilities. In general, I acknowledge the contribution of this paper on efficient and accurate operator learning by kernel tensorization. There're two more points I'd like to make:

  1. Regarding the Cartesian Grids, the traditional limitation of kernel type learning operators (FNO, etc.) may be alleviated by a projection from irregular physical space to regular grids on virtual (computational) space, for example, [1], [2].

  2. Also, there should be another important baseline, the Riesz Neural Operator [3], which should be directly comparable to your work.

[1] Cheng, Z., Li, Z., Wang, X., Huang, J., Zhang, Z., Hao, Z., & Su, H. (2025). Accelerating PDE-constrained optimization by the derivative of neural operators.

[2] Wu, H., Luo, H., Wang, H., Wang, J., & Long, M. (2024). Transolver: A fast transformer solver for pdes on general geometries. arXiv preprint arXiv:2402.02366.

[3] Liu, S., Yang, X., & Chen, Y. (2026). Riesz Neural Operator for Solving Partial Differential Equations. In The Fourteenth International Conference on Learning Representations.

AuthorsOpenReview ↗

Follow-up Response to Reviewer FKU4

Thank you for your further feedback with sharing these useful references. We agree that projecting data from irregular physical domains onto regular computational grids can help extend Cartesian-grid neural operators to complex geometries. Following your suggestion, we have also added the RNO baseline.

RNO is closely related to our method: both exploit directional structure to approximate PDE solution operators. RNO mixes directional Riesz components within multidimensional spectral layers, whereas our method composes independently parameterized Transformer blocks, each attending along a single coordinate. We therefore agree that RNO is a useful baseline for comparison. We compare the two methods on RB at three model scales with comparable parameter counts. Fig. 14 (page 31) shows lower one-step VRMSE for our tensorized Transformer at all three scales.

Thank the author for the quick response. The additional comparison with RNO is valuable. I have no further concern and therefore remain my score and recommend acceptance.

Reviewer_3gCxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a tensorized Transformer for PDE operator learning. Based on tensorization, it decomposes full attention on a Cartesian grid into alternating attention across multiple dimensions, thereby reducing computational complexity and achieving scalability. Empirically, the paper performs controlled architectural substitutions between full, windowed, and tensorized attention across four The Well benchmarks (TRL, AM, RB, MHD). The paper also compares against existing factorized PDE Transformer baselines—FactFormer, AViT, and MORPH.

Critical strengths

S1. Controlled empirical comparison of attention patterns for PDE surrogates.

Critical weaknesses

W1. This work is essentially an application of Axial Attention to the "The-Well" dataset and lacks novelty. The authors should conduct an in-depth analysis of the characteristics of continuous medium variations and design a factorization from a more "physics-based" perspective; this would be more meaningful than simply applying established methods from the computer vision domain.

W2. The paper is cluttered with excessive formal analysis that offers no real theoretical contribution; this unnecessarily complicates simple issues and makes the material harder for the reader to understand.

W3. As noted in the conclusion, the method is limited to Cartesian grids, which diminishes its value for practical applications involving large-scale unstructured grids.

W4. The FNO baseline is missing from the experiments.

Questions for authors

The complex formal description leaves me puzzled. Can the "tensorized transformer" proposed by the authors be intuitively understood as performing attention alternately across the various dimensions of the tensor?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3gCx

Thank you for the thoughtful review and helpful comments. We have carefully considered these suggestions and revised the paper accordingly. The revised content is temporarily appended to the end of the appendix, and we will incorporate the key content into the main text as space allows.


W1. Relation to Axial Transformer and motivation from PDEs

Thank you for this helpful comment. We understand the reviewer’s concern about the relation between our work and Axial Transformer [1] and sincerely appreciate this feedback. We agree that we should clarify (i) how our work differs from Axial Transformer and (ii) why our design has more in common with Axial Transformer than with other factorized models.

First, our method differs from Axial Transformer in its modeling process. We use a Vision Transformer (ViT) [2] backbone to realize our proposed tensorization, with standard self-attention in each Transformer block updating all tokens in parallel. Axial Transformer [1], which predates ViT, uses attention masks for autoregressive generation, producing spatial nodes row by row and pixel by pixel. Empirically, we evaluate both backbones on RB with the same encoder, decoder, and Transformer block settings. Our models achieve higher efficiency and lower VRMSE than Axial Transformer (Fig.10, page 30).

Second, we sincerely agree that PDE models should be designed from a more physics-based perspective. Our factorization starts from a kernel decomposition of PDE solution operators. The intuition behind residual connections and the need for standardized comparisons then motivate independently parameterized Transformer blocks, each attending along one dimension. We acknowledge that the resulting design shares elements with established CV methods, while we consider this design a more natural choice than other factorizations in the PDE context. Our controlled comparisons with FactFormer, AViT, and MORPH support this choice among different factorizations. Furthermore, we prove that our tensorization can approximate continuous solution operators, providing PDE-specific theoretical support that has not been established in previous factorization works.

Thank you again for this helpful comment. We have added this discussion and the related experiments to the appendix and will further revise the paper accordingly.

  • [1] Ho et al. Axial Attention in Multidimensional Transformers. arXiv preprint, 2019.
  • [2] Dosovitskiy et al. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.

W2. Clarity of the formal analysis

Thank you for this comment. We acknowledge that we have much formal analysis. We start from kernel decomposition of PDE solution operators, which motivates the factorization and its realization with Transformer blocks. We then provide an approximation proof. We will try to simplify it to make the material clearer.

W3. Cartesian grids and complex geometries

Thank you for pointing out this issue. We acknowledge that our work focuses on PDE modeling on regular grids, a limitation shared by earlier factorized methods and state-of-the-art foundation models such as Walrus [3] and Poseidon [4].

For complex geometries, we agree that additional designs are needed to handle unstructured grids. One practical extension is to encode unstructured inputs onto a regular latent grid, following GINO [5] or adapting UPT’s supernode encoding [6], and apply our tensorized module there. We leave this extension to future work.

  • [3] McCabe et al. Walrus: A Cross-domain Foundation Model for Continuum Dynamics. ICML, 2026.
  • [4] Herde et al. Poseidon: Efficient Foundation Models for PDEs. NeurIPS, 2024.
  • [5] Li et al. Geometry-Informed Neural Operator for Large-Scale 3D PDEs. NeurIPS, 2023.
  • [6] Alkin et al. Universal Physics Transformers: A Framework for Efficiently Scaling Neural Operators. NeurIPS, 2024.

W4. Comparison with FNO

Thank you for this suggestion. We acknowledge that our comparisons with existing models focus on factorized PDE models, omitting a general baseline to anchor performance. We have added FNO at three parameter sizes comparable to our three model sizes. The results in Fig.13 (left, page 30) of the revised appendix show that our method consistently outperforms FNO at each model size.

Question: Intuitive interpretation of the tensorized Transformer

We are sorry for the confusion caused by our formal description. The reviewer’s intuitive understanding of our tensorized Transformer is correct.


Again, we sincerely thank the reviewer for the helpful comments and suggestions for improving our paper. We appreciate the reviewer’s time and effort and would be grateful for any further thoughts on our responses.

Reviewer_3gCxOpenReview ↗

Thanks for the rebuttal

Thanks for the rebuttal. My concerns are not fully addressed, considering its limited novelty, limited to Cartesian grids. I will keep my initial rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We appreciate the reviewers’ thoughtful and helpful reviews and their time and effort in reviewing our paper. We have revised the paper based on their comments and suggestions, adding the following experiments and analyses:

  1. Axial Transformer comparison (Reviewer 3gCx)
  2. FNO baselines (Reviewer 3gCx)
  3. Zero-shot resolution transfer (Reviewer FKU4)
  4. Linear and low-rank full attention baselines (Reviewer eLZP)
  5. Different attention patterns with Nyström (Reviewer eLZP)
  6. Training and validation curves (Reviewer eLZP)
  7. Coordinate-coupling ablation (Reviewer eLZP)
  8. Window-size ablation (Reviewer eLZP)

The added content is presented on pages 28-29 of the revised PDF, with all related figures on page 30. We would be grateful for further feedback and discussion and sincerely appreciate the reviewers’ time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.