- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies tensorized Transformers for PDE operator learning. The model applies independently parameterized Transformer blocks along individual coordinates, allowing global interaction after a complete coordinate sweep. The paper provides a kernel-tensorization interpretation and a universal approximation result. Experiments compare full, windowed, and tensorized attention on four PDE benchmarks, including one three-dimensional benchmark, across three model sizes.
Critical strengths
-
The controlled comparisons with full and windowed attention use the same encoder, decoder, model configurations, and training settings. The results show a useful accuracy–cost trade-off across several PDEs and model sizes, supported by measured inference time and ten-step rollout errors.
-
The kernel-tensorization view provides a mathematical explanation for composing coordinate-wise attention. The approximation analysis establishes that this model class can approximate continuous solution operators under the stated assumptions.
Critical weaknesses
-
The controlled comparison does not include representative linear-complexity or low-rank attention methods. Although the paper compares against FactFormer, A ViT, and MORPH, these retain their own architectures rather than serving as attention replacements within the same framework. It remains unclear whether tensorized attention offers a better accuracy–cost trade-off than alternatives such as kernel-based linear attention or Nyström attention. Adding one or two representative baselines with the same encoder, decoder, and training setup, and reporting complexity, runtime, peak memory, and prediction error, would strengthen the evaluation.
-
The experiments provide limited evidence for why tensorized attention improves accuracy. The approximation theorem establishes expressivity with sufficient capacity, but does not explain the gains at the finite model sizes and training budgets used here. The head-count experiment in Figure 8 offers indirect support for the hypothesis that one-dimensional factors are easier to learn, but does not distinguish better optimization, reduced overfitting, or a more suitable spatial inductive bias. For windowed attention, the difference may instead come from access to global interactions after one sweep. Training and validation curves, together with targeted ablations of coordinate coupling or window size, would help clarify these explanations.
Questions for authors
1.How does tensorized attention compare with a representative kernel-based linear attention or Nyström approximation under the same framework and training setup, considering both prediction error and measured computational cost?
2.Does full attention have higher training error, or mainly a larger gap between training and validation error? Can controlled changes to coordinate coupling and window size help identify the source of the accuracy gains?
2 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors’ efforts, which have substantially improved the paper.
However, I still find the central motivation insufficiently supported. While the new experiments demonstrate empirical improvements, they do not convincingly explain why tensorized attention outperforms full attention. A deeper analysis of the attention patterns and underlying mechanisms is still needed.
Therefore, I maintain my original rating of 2 (Weak Rejection).
Response to Reviewer eLZP
Thank you for your helpful review. We sincerely appreciate your time and effort and have revised our paper based on your constructive suggestions. The revised content is temporarily appended to the end of the appendix, and we will incorporate the key content into the main text as space allows.
W1 and Q1: Linear and Nyström attention
Thank you for this insightful suggestion. We agree that adding linear-complexity and low-rank attention strengthens the comparisons. On RB, we use Performer [7] as a linear-complexity baseline and Nyström attention [8] as a low-rank baseline for full attention. Fig.11 (left, page 30) shows that our tensorized method achieves lower prediction error at comparable computational costs.
Inspired by the reviewer’s suggestion, we further expand the comparisons. Full, windowed, and tensorized attention are three different attention patterns, whereas softmax, linear, and Nyström attention are different attention implementations. Each implementation can be used with all three patterns. Fig.11 (right, page 30) compares the accuracy and computational cost of the three patterns implemented with Nyström attention. Tensorized attention consistently achieves lower prediction error than the other two patterns. The VRMSE gap between full and tensorized attention is larger than with softmax attention.
The latter results also support our motivation that kernel decomposition can simplify the features to be learned. With a low-rank approximation, full attention may struggle to represent a complex kernel, while tensorized and windowed attention can remain effective with simpler kernel structures.
We sincerely thank the reviewer for this suggestion, which also helps clarify why tensorized attention can be more effective than full attention.
W2: Limited evidence for why tensorized attention improves accuracy
Thank you for pointing out this limitation of our paper. We acknowledge that our original experiments provide limited evidence for the hypothesis that decomposed kernel factors are easier to learn. The low-rank attention comparisons suggested in W1 help clarify this motivation, as discussed above. We will use these experimental results to strengthen the discussion in the main text.
W2 and Q2: Training and validation curves
Thank you for this helpful suggestion. We have added training and validation curves for models on RB and MHD as two representative samples (Fig.12, page 30). Tensorized attention achieves lower final training MSE and validation VRMSE in both settings. This supports improved fitting under the same training budget, consistent with our hypothesis that decomposed kernel factors are easier to learn.
W1, W2, and Q2: Coordinate-coupling ablation
Thank you for reminding us of this important ablation. In our tensorized method, we sequentially couple coordinates by composing consecutive Transformer blocks, each attending along a different coordinate. This is a major difference from other factorizations like AViT and MORPH, which couple coordinates in parallel and combine the outputs by averaging or summation. We therefore ablate the way coordinates are coupled to (i) clarify the benefit of our coordinate coupling and (ii) strengthen the comparison with parallel factorizations.
The results in Fig.13 (right, page 30) show a consistent accuracy gain from our sequential coupling over parallel coupling. Experimental details are provided alongside the figure.
W2 and Q2: Window-size ablation
Thank you for reminding us of this important ablation. We ablate the window size of the large windowed attention models. Table 13 (page 29) summarizes the window-size ablation results.
Again, we thank the reviewer for the helpful comments and suggestions that have improved our paper. We appreciate the reviewer’s time and effort and would be grateful for any further thoughts on our responses.