- Rating
- 4(+3)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes physics-attention, the core layer of Transolver.
On the theory side, the paper makes three claims:
A stack with a constant core is universal (Prop. 1), shown by reduction to the averaging neural operator (ANO) of Lanthaler et al. (2023). Pointwise stacks cannot approximate nonlocal operators (Cor. 1). An upper bound limits how many nonlocal functionals pass through one coupling (Prop. 2).
Finally, the paper presents FlashSlice, fused Triton kernels that never materialize the N×G slice-weight tensor.
Critical strengths
A controlled ablation at industrial scale. All variants share one training pipeline across nine large 3D aerodynamic and turbulence configurations, with up to 1.3×10⁸ points per sample. Each ablation is checked mechanically through checkpoint-key and bit-level freeze tests (App. C.2). The token-attention ablation stays at about 1×, while the coupling ablations reach up to 5.6× (MlpOnly) and 6.8× (SliceOnce). This gap is consistent in direction across all nine configurations and gives practitioners a usable signal about a widely deployed architecture.
A carefully validated kernel. FlashSlice is checked against an fp64 reference, is bit-deterministic, and is benchmarked against compiled and checkpointed baselines (Tables F.4, F.5). Its memory use is flat in G (Table F.3), which is a practical benefit for large-mesh training.
Critical weaknesses
(a) The empirical finding repeats Hu et al. (2026). The headline result is that token attention inside physics-attention can be removed without loss of accuracy. Hu et al. (2026, "Transolver is a Linear Transformer", AAAI) already reported this: they recast physics-attention as linear attention, removed the slice attention, and proposed LinearNO on that basis. The authors acknowledge this.
What this submission adds is mainly scale (nine 3D benchmarks) and a few more ablations of the coupling. Those additional results do not lead to a qualitatively new conclusion. "The attention is redundant; the global aggregation carries the accuracy" was already the message of prior work.
The one place where the paper takes a different position is its disagreement with Hu et al. on why the attention is redundant (projection tying). That argument rests on a single 2×2 grid on one benchmark (Fig. 2a), in which all four cells lie within about 1% of each other.
(b) The theory does not explain the experiments it is meant to explain.
Prop. 1 is a direct application of an existing theorem. It applies the ANO universality theorem of Lanthaler et al. (2023): setting all slice logits to zero makes every token the global mean, so the result follows. The construction does not use the slice structure, the learned partition of unity, or G at all. It would apply equally to any architecture that contains a global mean and pointwise MLPs.
Prop. 1 cannot separate the models being compared. As the authors themselves say (App. A.2, L756–764), the class with attention and the class without it are both universal. The result cannot distinguish BASELINE from NoTokenAttn, which is the comparison the paper is about.
Cor. 1 is close to trivial. It says that a composition of pointwise maps is pointwise.
Prop. 2 is loose by the authors' own account. It is a counting bound that the authors describe as "rather loose" (L328), and they state that it does not explain the size of the SliceOnce penalty (L821–829).
The links to the experiments are mostly hedged and qualitative. Examples include "suggestive of some of the other observations" (L318), "makes this observation unsurprising" (L302), and "the magnitude is not something we derive" (L829).
Questions for authors
- Will the FlashSlice code be made available to reviewers during the discussion period?
2.Apart from the slice/deslice operator of Transolver, what is new in the kernel design compared with FlashAttention and the tiled recomputation in Transolver-3?
8 comments
Thank you for the response.
- Novelty. The main finding, that token attention can be removed without loss, was already reported by Hu et al. (2026). The new tying experiment addresses only a secondary point.
- Theory. The authors accept all four points, and the new runs do not change this. Cor. 1 holds by construction. Prop. 2 is an upper bound, consistent with both a flat and a monotone K-sweep. Prop. 1 does not predict trained accuracy.
- Kernel. The comparison with Transolver-3 is new to the rebuttal and covers a single layer at one size. Inside the tiling, FlashSlice gives a 2.5× speedup but does not lower the memory floor, so it does not extend the trainable mesh size. This is solid engineering, but not a new idea.
The revision would substantially change the paper's claims, so I maintain my rating.
Response to Reviewer GFrv's follow-up (1/2): Novelty and Theory
We thank the reviewer for their prompt reply and respect the opinion of the reviewer. However, we wish to rebut the points that you have raised below:
- Novelty: We had already acknowledged in the submitted version of our paper that the Hu et. Al. had pointed out that attention is not needed in Transolver and we have repeated this point multiple times in our rebuttal. However, their demonstration was on very small-scale 2-D experiments and lacked a coherent explanation of why this is the case. Our paper has been written to precisely address these limitations. We consider large-scale 3D industrial benchmarks and find that token-attention is not necessary. More crucially, we have proposed a coherent explanation of why token attention can be removed without any loss of accuracy. We have used both empirical as well as theoretical means to do so and these provide a much deeper explanation to the reader as to what is essential in the Transolver model and what is not. Interpreting the reviewer’s conclusion as “once a phenomenon is observed, there is no further need to investigate and explain it” runs counter to the standard practice in science where the purpose of many further studies is to precisely find such explanations of new observations. We would appreciate if the reviewer points out genuine issues with our explanation but dismissing them as secondary does not constitute a point that we can constructively engage with.
- Theory: We have repeatedly stated that Proposition 1 describes the limits of what is expressible by a certain neural operator model. Using it, clearly shows that token attention is not necessary for expressing relevant classes of operators and adding or removing token attention should not change what operators a model can approximate and what it cannot. We believe that this already provides an explanation of why token attention is redundant. Asking for precise bounds on trained models in this context is unrealistic as no such bounds exist in the literature, to the best of our knowledge. We would be happy to add references to such precise estimates if the reviewer can kindly provide them. Moreover, we would like to reemphasize that, although it is simple in its formalism and calculations, our theory explains several aspects of the observed empirical results, providing the reader with a coherent explanation resting on empiricism, heuristics and theory. We believe that such an approach is fruitful in this domain, rather than looking for precise estimates which are unrealistic to obtain, given current tools.
Thank you for a detailed and precise review. We agree with much of it.
(a) The empirical finding and Hu et al. (2026)
You are correct in saying that our disagreement with Hu et al. rested on a single grid. We extended the 2×2 factorial of attention × tying to a second benchmark (Table A of the general response). Every cell again lies within 0.7% of the Baseline, so the conclusion of the paper stands: tying is orthogonal to the redundancy of the attention.
The other differences from Hu et al. are listed in item 1 of the general response.
(b) The theory
We accept your four points: Proposition 1 applies the ANO theorem directly, it cannot separate Baseline from NoTokenAttn (as Appendix A.2 states) nor is it inteneded to, Corollary 1 is elementary, and Proposition 2 is a loose counting bound. Our original presentation could potentially be misread in how the theory explains the numerical evidence. In the revision we will:
Within that scope, the theory makes three predictions, which the new runs test:
Q1: Code availability
Yes. An anonymized repository is available at https://anonymous.4open.science/r/flashslice-review-557F/. It contains the FlashSlice kernels, the layer with every ablation flag, the tests and the benchmark scripts. We will release the code publicly on acceptance.
Q2: What is new in the kernel?
Against FlashAttention:
Against Transolver-3:
their tiling is a schedule in PyTorch that trades time for memory. FlashSlice is a fused kernel that lowers both. The two do not conflict: the kernels can run inside each tile. One layer, forward and backward, at 4.2M points:
Inside the tiling, FlashSlice keeps the tiling's memory at 2.5× its speed, and matches the eager layer to 6×10⁻⁷ (output) and 4×10⁻⁶ (gradients), relative. We will add this comparison to Section 5.