- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes GTNO, a neural operator that uses a high-resolution grid for spatial modeling and compact tokens for temporal modeling. The two representations exchange information through bidirectional interactions and spectral corrections. Experiments on five PDE benchmarks show improved prediction accuracy over the evaluated baselines. The authors also provide ablation studies and efficiency comparisons to validate the proposed design.
Critical strengths
-
The proposed Grid–Token architecture provides a meaningful methodological contribution. In particular, the asymmetric bidirectional interaction between spatial grids and temporal tokens is a novel and well-motivated design.
-
The experimental results are strong. The method achieves the best prediction accuracy across five PDE benchmarks, and the ablation studies provide convincing evidence for the effectiveness of the proposed architecture.
Critical weaknesses
-
I think the unique value of the Spectral Bridge needs further validation. In Table 7, removing Token-to-Grid also removes the entire Spectral Bridge. Therefore, it is unclear whether the improvement comes from the temporal tokens or simply from the Fourier correction. A more direct comparison would help clarify this.
-
The necessity of bidirectional interaction also needs further explanation. For example, removing Grid-to-Token has little impact on NS2D, while removing Token-to-Grid significantly reduces performance. I think the authors should explain why both directions are needed and provide more direct evidence of their benefits.
Questions for authors
Please refer to the Critical Weaknesses section.
8 comments
Thank you for your detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.
After reading the paper more carefully, I think the experimental evaluation could be further strengthened to better support the main contributions.
First, since explicit temporal modeling is one of the key contributions of GTNO, I think more direct comparisons with temporal modeling methods, such as MemNO and ENMA, would make the results more convincing.
Second, I would like to see evaluations on more challenging temporal dynamics, such as Kolmogorov flow (KF) or Kuramoto–Sivashinsky (KS), as well as settings with noisy or partial observations. These tasks could help demonstrate the benefits of the proposed temporal modeling mechanism.
Finally, I would appreciate a clearer explanation of why disabling cross-frame attention improves performance for training windows of 3, 5, and 7.
I understand that additional experiments may be difficult during the rebuttal period. However, I believe these points would help make the experimental evidence more solid and better support the proposed contributions.
Reviewer 3
Thank you for your thoughtful and constructive comments. We sincerely apologize for our delayed response, as we needed additional time to conduct and organize further experiments to thoroughly address your concerns. We greatly appreciate your patience and understanding. We provide our point-by-point responses below.
R3 - Q3
Q3: Direct comparisons with temporal modeling methods. Since explicit temporal modeling is a key contribution of GTNO, more direct comparisons with methods such as MemNO and ENMA would make the results more convincing.
A3: Thank you for this suggestion. We have added comparisons with MemNO and ENMA on five benchmarks, matching the observed history, prediction targets, and full-rollout relative evaluation protocol across methods.
Full-rollout relative error (↓ lower is better). The best result on each dataset is highlighted in bold.
| Model / Dataset | FNO NS2D | DR2D | CNS2D | SWE2D | PDEArena NS2D |
|---|---|---|---|---|---|
| GTNO | 0.045 | 0.007 | 0.098 | 0.000693 | 0.282 |
| MemNO | 0.051736 | 0.016849 | 0.121128 | 0.002709 | 0.335783 |
| ENMA | 0.232116 | 0.975454 | 0.561667 | 0.059713 | 0.570410 |
MemNO. Since the official implementation of MemNO is not publicly available, we independently reproduced its S4FFNO variant based on the architecture and main configurations described in the original paper. Under the same observation windows, prediction targets, and full-rollout relative error metric, GTNO consistently achieves lower errors than our MemNO reproduction across all five benchmarks. Specifically, GTNO reduces the prediction error by 13.0%, 58.5%, 19.1%, 74.4%, and 16.0% on FNO NS2D, DR2D, CNS2D, SWE2D, and PDEArena NS2D, respectively, corresponding to an average error reduction of 36.2%.
ENMA. ENMA is a flow-matching-based generative model that relies on iterative generation and sampling during inference. We evaluated its publicly available official implementation using the released configurations, following its two-stage training procedure with 500 epochs of VAE pretraining and 1,000 epochs of predictor training. The reported results are averaged over three generation seeds with the trained weights fixed. Under the adopted configuration, GTNO achieves lower full-rollout prediction errors than ENMA across all five benchmarks. However, a strictly fair comparison is difficult because ENMA is a generative model whose performance depends on the number of generation iterations and sampling settings, which can substantially affect the final prediction results. Therefore, we report this comparison for reference only.
Nevertheless, these comparisons provide additional empirical evidence of GTNO's effectiveness in temporal prediction.
R3 - Q4
Q4: Challenging temporal dynamics and imperfect observations. Evaluations on Kolmogorov flow or Kuramoto--Sivashinsky dynamics, and on noisy or partial observations, would help demonstrate the benefits of the temporal modeling mechanism.
[1] Kochkov et al. "Machine learning–accelerated computational fluid dynamics." PNAS, 2021.
A4: We agree and have added a Kolmogorov flow (KF) evaluation, including long autoregressive forecasts and robustness tests in which the initial observations are corrupted by noise.
1. Long-horizon Kolmogorov flow. We use the Kochkov/JAX-CFD KF data at on a grid, predicting the raw velocity components . The evaluation uses 16 held-out trajectories, with five non-overlapping prediction windows per trajectory, yielding 80 evaluation windows. All models begin with 10 ground-truth observations and predict 300 future steps without ground-truth feedback. GTNO uses an inference configuration consistent with its temporal training setup. The sampling interval is fixed at , so the 300-step rollout spans 21.04 simulation time units.
Prefix relative error (↓ lower is better).
The table reports full-rollout relative error up to each forecast horizon. The best result at each horizon is highlighted in bold.
| Model / Setting | H = 10 | H = 50 | H = 100 | H = 200 | H = 300 |
|---|---|---|---|---|---|
| Simulated duration | 0.7012 | 3.5062 | 7.0125 | 14.0250 | 21.0375 |
| GTNO | 0.005434 | 0.033639 | 0.183658 | 0.778816 | 1.014027 |
| Transolver | 0.566274 | 1.169816 | 1.225469 | 1.233711 | 1.237234 |
| LaMO | 0.013603 | 0.114697 | 0.418278 | 0.937240 | 1.106575 |
| SAOT | 0.019213 | 0.178475 | 0.537448 | 0.983968 | 1.132088 |
GTNO achieves the lowest cumulative relative error at every reported horizon. To distinguish cumulative prediction accuracy from sustained dynamical fidelity, we also evaluate vorticity correlation.
R3 — Q1
Thank you for raising this concern. We agree that the original ablation does not isolate the contributions of temporal information and Fourier correction. To further investigate the role of temporal information, we evaluated models trained with different temporal windows using a Self-only attention mask at inference, which blocks cross-frame attention while retaining the Spectral Bridge and its Fourier correction.
On NS2D, we report relative $ L_2 $ error over 10 autoregressive prediction steps, averaged across available seeds. Full uses the same temporal window as during training, while Self-only disables cross-frame attention at inference.
At $ W = 10 $, blocking cross-frame attention increases error by 123.67% despite retaining Fourier correction, indicating that temporal information plays an important role in the model's performance at this window size. In contrast, the model achieves competitive performance at $ W = 1 $, where no cross-frame attention is available.
Interestingly, Self-only improves performance at intermediate windows ($ W = 3, 5, 7 $). This may suggest that the model does not always effectively exploit the available historical information at smaller training windows, whereas Self-only introduces a Markov-like inductive bias.
Overall, these results indicate that the importance of temporal interactions depends on the training window and becomes substantial at $ W = 10 $. Combined with the original ablation results, the performance gains of the Spectral Bridge cannot be simply attributed to Fourier correction alone.
R3 — Q2
Thank you for this observation. The two directions are asymmetric in both their roles and computational costs. Grid-to-Token refreshes the compact token representations with updated spatial information through lightweight convolutions and learned downsampling, while Token-to-Grid applies token-conditioned spectral residual corrections to the grid.
Importantly, tokens already receive spatial information at initialization. Thus, removing Grid-to-Token eliminates repeated spatial updates rather than all spatial information, which may explain its relatively small impact on NS2D.
To further examine its role, we analyzed the learned Grid-to-Token fusion gates during autoregressive evaluation:
The mixing weights are averaged across samples, blocks, spatial positions, channels, and rollout steps. These relatively high weights suggest that the model learns to substantially incorporate information from the grid branch. Both interaction directions employ gated fusion, allowing the model to adaptively regulate cross-branch information exchange. Furthermore, the ablation results show that removing Grid-to-Token increases error by 4.4% on NS2D and 19.9% on SWE2D, demonstrating its task-dependent benefits.
We also profile the computational costs of both interaction modules within the full model:
Overall, the two directions have asymmetric roles and computational costs. Grid-to-Token continuously refreshes spatial information in the token representations, improving performance on both NS2D (4.4%) and SWE2D (19.9%), while requiring only about one-tenth of the profiled operations and one-eighth of the parameters of Token-to-Grid. These results support retaining both directions, with complementary functions and task-dependent benefits.