acceptodds
Under review as a conference paper at ICLR 2027

Grid-Token Neural Operator: Asymmetric Spatiotemporal Coupling for PDE Forecasting

Abstract

Neural operators offer an efficient surrogate for repeated numerical simulation of partial differential equations (PDEs), but jointly representing spatial detail and temporal dependencies remains challenging. Applying temporal attention at every fine-grid location is costly, whereas compressing the entire evolving state places spatial reconstruction and temporal modeling behind the same bottleneck. To address this trade-off, we propose the Grid-Token Neural Operator (GTNO), which maintains two complementary representations throughout the network: a structured grid for spatial refinement and compact tokens for causal temporal modeling. Each layer couples time-shared local grid updates with token-wise temporal attention through bidirectional interaction. Local grid-to-token routing refreshes the temporal representation, while token-conditioned spectral corrections feed temporal context back to the grid. This asymmetric design concentrates expensive causal sequence modeling on compact tokens while retaining a dedicated spatial pathway. Across five time-dependent PDE benchmarks, GTNO consistently outperforms eight neural operator baselines, demonstrating the effectiveness and efficiency of combining fine spatial evolution with compact temporal modeling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Time-dependent PDEs, Spatio-Temporal Modeling, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_61MpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes GTNO, a neural operator that uses a high-resolution grid for spatial modeling and compact tokens for temporal modeling. The two representations exchange information through bidirectional interactions and spectral corrections. Experiments on five PDE benchmarks show improved prediction accuracy over the evaluated baselines. The authors also provide ablation studies and efficiency comparisons to validate the proposed design.

Critical strengths
  1. The proposed Grid–Token architecture provides a meaningful methodological contribution. In particular, the asymmetric bidirectional interaction between spatial grids and temporal tokens is a novel and well-motivated design.

  2. The experimental results are strong. The method achieves the best prediction accuracy across five PDE benchmarks, and the ablation studies provide convincing evidence for the effectiveness of the proposed architecture.

Critical weaknesses
  1. I think the unique value of the Spectral Bridge needs further validation. In Table 7, removing Token-to-Grid also removes the entire Spectral Bridge. Therefore, it is unclear whether the improvement comes from the temporal tokens or simply from the Fourier correction. A more direct comparison would help clarify this.

  2. The necessity of bidirectional interaction also needs further explanation. For example, removing Grid-to-Token has little impact on NS2D, while removing Token-to-Grid significantly reduces performance. I think the authors should explain why both directions are needed and provide more direct evidence of their benefits.

Questions for authors

Please refer to the Critical Weaknesses section.

8 comments

R3 — Q1

Thank you for raising this concern. We agree that the original ablation does not isolate the contributions of temporal information and Fourier correction. To further investigate the role of temporal information, we evaluated models trained with different temporal windows using a Self-only attention mask at inference, which blocks cross-frame attention while retaining the Spectral Bridge and its Fourier correction.

On NS2D, we report relative $ L_2 $ error over 10 autoregressive prediction steps, averaged across available seeds. Full uses the same temporal window as during training, while Self-only disables cross-frame attention at inference.

Training window $ W $FullSelf-onlyError change
10.0517340.0517340.00%
30.1397350.047529−65.99%
50.0965650.048718−49.55%
70.0829220.056227−32.19%
100.0453560.101446+123.67%

At $ W = 10 $, blocking cross-frame attention increases error by 123.67% despite retaining Fourier correction, indicating that temporal information plays an important role in the model's performance at this window size. In contrast, the model achieves competitive performance at $ W = 1 $, where no cross-frame attention is available.

Interestingly, Self-only improves performance at intermediate windows ($ W = 3, 5, 7 $). This may suggest that the model does not always effectively exploit the available historical information at smaller training windows, whereas Self-only introduces a Markov-like inductive bias.

Overall, these results indicate that the importance of temporal interactions depends on the training window and becomes substantial at $ W = 10 $. Combined with the original ablation results, the performance gains of the Spectral Bridge cannot be simply attributed to Fourier correction alone.

R3 — Q2

Thank you for this observation. The two directions are asymmetric in both their roles and computational costs. Grid-to-Token refreshes the compact token representations with updated spatial information through lightweight convolutions and learned downsampling, while Token-to-Grid applies token-conditioned spectral residual corrections to the grid.

Importantly, tokens already receive spatial information at initialization. Thus, removing Grid-to-Token eliminates repeated spatial updates rather than all spatial information, which may explain its relatively small impact on NS2D.

To further examine its role, we analyzed the learned Grid-to-Token fusion gates during autoregressive evaluation:

DatasetMean grid-candidate mixing weight
NS2D65.1%
SWE2D71.6%

The mixing weights are averaged across samples, blocks, spatial positions, channels, and rollout steps. These relatively high weights suggest that the model learns to substantially incorporate information from the grid branch. Both interaction directions employ gated fusion, allowing the model to adaptively regulate cross-branch information exchange. Furthermore, the ablation results show that removing Grid-to-Token increases error by 4.4% on NS2D and 19.9% on SWE2D, demonstrating its task-dependent benefits.

We also profile the computational costs of both interaction modules within the full model:

Cost metric (per block)Grid-to-Token moduleToken-to-Grid module
Parameters (including fusion gate)95,232733,568
Operations (per sample per time step)84.1 MFLOPs873.9 MFLOPs

Overall, the two directions have asymmetric roles and computational costs. Grid-to-Token continuously refreshes spatial information in the token representations, improving performance on both NS2D (4.4%) and SWE2D (19.9%), while requiring only about one-tenth of the profiled operations and one-eighth of the parameters of Token-to-Grid. These results support retaining both directions, with complementary functions and task-dependent benefits.

Thank you for your detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.

After reading the paper more carefully, I think the experimental evaluation could be further strengthened to better support the main contributions.

First, since explicit temporal modeling is one of the key contributions of GTNO, I think more direct comparisons with temporal modeling methods, such as MemNO and ENMA, would make the results more convincing.

Second, I would like to see evaluations on more challenging temporal dynamics, such as Kolmogorov flow (KF) or Kuramoto–Sivashinsky (KS), as well as settings with noisy or partial observations. These tasks could help demonstrate the benefits of the proposed temporal modeling mechanism.

Finally, I would appreciate a clearer explanation of why disabling cross-frame attention improves performance for training windows of 3, 5, and 7.

I understand that additional experiments may be difficult during the rebuttal period. However, I believe these points would help make the experimental evidence more solid and better support the proposed contributions.

Reviewer 3

Thank you for your thoughtful and constructive comments. We sincerely apologize for our delayed response, as we needed additional time to conduct and organize further experiments to thoroughly address your concerns. We greatly appreciate your patience and understanding. We provide our point-by-point responses below.

R3 - Q3

Q3: Direct comparisons with temporal modeling methods. Since explicit temporal modeling is a key contribution of GTNO, more direct comparisons with methods such as MemNO and ENMA would make the results more convincing.

A3: Thank you for this suggestion. We have added comparisons with MemNO and ENMA on five benchmarks, matching the observed history, prediction targets, and full-rollout relative L2L_2 evaluation protocol across methods.

Full-rollout relative L2L_2 error (↓ lower is better). The best result on each dataset is highlighted in bold.

Model / DatasetFNO NS2DDR2DCNS2DSWE2DPDEArena NS2D
GTNO0.0450.0070.0980.0006930.282
MemNO0.0517360.0168490.1211280.0027090.335783
ENMA0.2321160.9754540.5616670.0597130.570410

MemNO. Since the official implementation of MemNO is not publicly available, we independently reproduced its S4FFNO variant based on the architecture and main configurations described in the original paper. Under the same observation windows, prediction targets, and full-rollout relative L2L_2 error metric, GTNO consistently achieves lower errors than our MemNO reproduction across all five benchmarks. Specifically, GTNO reduces the prediction error by 13.0%, 58.5%, 19.1%, 74.4%, and 16.0% on FNO NS2D, DR2D, CNS2D, SWE2D, and PDEArena NS2D, respectively, corresponding to an average error reduction of 36.2%.

ENMA. ENMA is a flow-matching-based generative model that relies on iterative generation and sampling during inference. We evaluated its publicly available official implementation using the released configurations, following its two-stage training procedure with 500 epochs of VAE pretraining and 1,000 epochs of predictor training. The reported results are averaged over three generation seeds with the trained weights fixed. Under the adopted configuration, GTNO achieves lower full-rollout prediction errors than ENMA across all five benchmarks. However, a strictly fair comparison is difficult because ENMA is a generative model whose performance depends on the number of generation iterations and sampling settings, which can substantially affect the final prediction results. Therefore, we report this comparison for reference only.

Nevertheless, these comparisons provide additional empirical evidence of GTNO's effectiveness in temporal prediction.

R3 - Q4

Q4: Challenging temporal dynamics and imperfect observations. Evaluations on Kolmogorov flow or Kuramoto--Sivashinsky dynamics, and on noisy or partial observations, would help demonstrate the benefits of the temporal modeling mechanism.

[1] Kochkov et al. "Machine learning–accelerated computational fluid dynamics." PNAS, 2021.

A4: We agree and have added a Kolmogorov flow (KF) evaluation, including long autoregressive forecasts and robustness tests in which the initial observations are corrupted by noise.

1. Long-horizon Kolmogorov flow. We use the Kochkov/JAX-CFD KF data at Re=1000Re=1000 on a 64×6464\times64 grid, predicting the raw velocity components (u,v)(u,v). The evaluation uses 16 held-out trajectories, with five non-overlapping prediction windows per trajectory, yielding 80 evaluation windows. All models begin with 10 ground-truth observations and predict 300 future steps without ground-truth feedback. GTNO uses an inference configuration consistent with its temporal training setup. The sampling interval is fixed at Δt=0.070124836\Delta t=0.070124836, so the 300-step rollout spans 21.04 simulation time units.

Prefix relative L2L_2 error (↓ lower is better).

The table reports full-rollout relative L2L_2 error up to each forecast horizon. The best result at each horizon is highlighted in bold.

Model / SettingH = 10H = 50H = 100H = 200H = 300
Simulated duration0.70123.50627.012514.025021.0375
GTNO0.0054340.0336390.1836580.7788161.014027
Transolver0.5662741.1698161.2254691.2337111.237234
LaMO0.0136030.1146970.4182780.9372401.106575
SAOT0.0192130.1784750.5374480.9839681.132088

GTNO achieves the lowest cumulative relative L2L_2 error at every reported horizon. To distinguish cumulative prediction accuracy from sustained dynamical fidelity, we also evaluate vorticity correlation.

Reviewer_HaN8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Grid–Token Neural Operator (GTNO) for time-dependent PDE forecasting. The model maintains two representations: a structured grid for local spatial refinement and compact tokens for causal temporal modeling. Each layer couples these representations through bidirectional interaction. A learned grid-to-token router updates the tokens with spatial features, while token-to-grid cross-attention uses Fourier-mode queries and temporally informed tokens to generate spectral corrections to the grid. Training uses causally masked, shifted next-state supervision, and inference proceeds autoregressively with key–value caching. The experiments compare GTNO with eight neural-operator baselines on five two-dimensional PDE benchmarks, reporting the lowest mean full-rollout relative L2 error on each benchmark. Additional experiments examine component contributions, interaction directions, representation resolutions, temporal ordering, and computational costs.

Critical strengths
  1. Competitive predictive performance across several PDE benchmarks.

Table 1 reports the lowest mean full-rollout relative L2 error on all five datasets. The improvements over the strongest evaluated baseline are substantial on NS2D, DR2D, and SWE2D, at approximately 27.4%, 30.0%, and 28.6%, respectively. Table 6 also provides run-to-run statistics for GTNO and the strongest competing baseline on each dataset. These results provide meaningful evidence for the effectiveness of the complete system across different PDE families.

  1. A useful architectural division between spatial refinement and temporal modeling.

Maintaining a structured grid while applying causal temporal attention to compact tokens is a coherent approach to balancing spatial representation and temporal computation. The resolution study in Figure 5(a) provides useful empirical support: on NS2D-PA, performance is more sensitive to grid resolution than to token resolution. This offers a practical design insight into allocating different spatial resolutions to the two branches.

Critical weaknesses
  1. The central coupling mechanism is not sufficiently isolated by the ablations.

Appendix A.2.7 explicitly states that removing token-to-grid removes the entire spectral bridge. The resulting degradation therefore combines the loss of temporal conditioning with the loss of the global spatial operation provided by that bridge. For example, the NS2D error in Table 7 increases from 0.045 to 0.422 without token-to-grid, whereas removing grid-to-token changes it to 0.047. SWE2D shows a larger effect of removing grid-to-token, but Table 7 does not report variability across independent training runs. These results establish the importance of the implemented pathway, but do not establish how much of the gain specifically requires repeated bidirectional temporal coupling. This distinction matters because that coupling is a central methodological contribution. A control that preserves the spectral computation while removing historical information from the temporal tokens would be considerably more informative.

  1. Architectural changes are entangled with changes in the training formulation.

Appendix A.2.4 describes GTNO training on W+1-frame segments with causally masked, shifted next-state supervision. The fixed-window models instead use 2W-frame segments and W autoregressive training steps with prediction feedback. Grid-only also switches to this fixed-window formulation. The authors acknowledge in Appendix A.2.7 that these are comparisons of complete systems rather than controlled tests of temporal conditioning. Nevertheless, this leaves the architectural interpretation of the gains unresolved. The per-epoch training-cost comparison in Figure 4 likewise reflects the different training procedures as well as the architectures. A competitive spatial or bottleneck-memory baseline evaluated under a shared temporal training objective would help establish whether the proposed coupling provides an advantage beyond the change in optimization. Without such a comparison, I find the evidence for the specific architectural contribution insufficient for acceptance.

Questions for authors
  1. Requested evidence to resolve the coupling concern: Could you retrain a control that retains the spectral bridge and spatial routing but restricts temporal attention to the current frame in every layer, for example through a self-only temporal mask? This would preserve the global spectral pathway while removing historical token information. Does the full model retain a clear advantage? Please also report independent-seed statistics for the directional ablations, particularly the 0.045 versus 0.047 NS2D comparison.

  2. Requested evidence to resolve the training-protocol concern: Does GTNO retain its advantage over a competitive spatial or bottleneck-memory baseline when both use the same temporal training objective, with matched observed history, supervised target positions, and comparable training budgets? A comparison on one representative fluid benchmark would address this concern. How much of the training-time difference remains under that matched protocol?

  3. Long-horizon autoregressive forecasting. Table 4 evaluates NS2D and NS2D-PA over 10 predicted steps and CNS2D over 11, while DR2D and SWE2D include 91-step forecasts. Can GTNO maintain useful predictive accuracy over a few hundred autoregressive steps, for example 200–500 [1], on a Navier–Stokes benchmark with sufficiently long reference trajectories? Please keep the evaluation time interval fixed and report the corresponding physical duration. Per-step error or correlation, together with a spectral diagnostic, would help distinguish the useful prediction horizon from longer-term physical behavior. This would substantiate the long-horizon discussion in Section 4.3, particularly given the importance of low-amplitude spatial frequencies for long-rollout accuracy demonstrated by PDE-Refiner [2].

  4. Applicability beyond two-dimensional simulated benchmarks. All experiments concern two-dimensional simulated PDEs, and Appendix A.1.1 explicitly leaves the 3D implementation and evaluation to future work. Beyond the implementation changes already identified there, what are the main computational or modeling obstacles to applying GTNO to 3D Navier–Stokes forecasting? Do the authors have evidence that the proposed grid–token separation offers a useful accuracy–cost trade-off in this regime, or improves prediction on experimentally measured flow fields, such as those in the real-data branch of RealPDEBench [3]? As additional validation, a representative comparison in either setting, reporting predictive accuracy, inference time, and peak memory against a competitive baseline, would help establish the practical scope of the method.

[1] Kochkov et al. “Machine learning–accelerated computational fluid dynamics.” PNAS, 2021.

[2] Lippe et al. “PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers.” NeurIPS, 2023.

[3] Hu et al. “RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data.” ICLR, 2026.

9 comments

Reviewer 2

Thank you for your valuable comments and constructive feedback. We apologize for the delayed response, as we needed additional time to complete an extensive set of experiments addressing your concerns. We appreciate your patience and provide our point-by-point responses below.

R2 - Q1

Q1: Requested evidence to resolve the coupling concern: Could you retrain a control that retains the spectral bridge and spatial routing but restricts temporal attention to the current frame in every layer, for example through a self-only temporal mask? This would preserve the global spectral pathway while removing historical token information. Does the full model retain a clear advantage? Please also report independent-seed statistics for the directional ablations, particularly the 0.045 versus 0.047 NS2D comparison.

A1: Thank you for this constructive suggestion. We conducted additional controls to examine the contribution of temporal context while retaining the Spectral Bridge and Fourier correction, named self-only experiments.

1. Self-only temporal mask

We trained a model with a context window of W=1W=1, which prevents temporal attention from accessing historical frames. This setting is equivalent to a self-only temporal attention mask in terms of cross-frame attention visibility. Separately, we evaluated the model trained with W=10W=10 using a self-only temporal attention mask at inference.

The table below reports NS2D relative L2L_2 error over 10 autoregressive prediction steps, averaged over the available seeds. At inference, standard inference uses the context window employed during training, whereas self-only inference blocks cross-frame attention while retaining Fourier correction.

Training window $ W $Standard inferenceSelf-only inferenceError change under self-only masking
10.0517340.0517340.00%
100.0453560.101446+123.67%

The full model trained with W=10W=10 achieves 12.3% lower error than the W=1W=1 control, supporting the benefit of a larger temporal context. Furthermore, applying self-only masking at inference to the W=10W=10 model increases its error by 123.67%, despite retaining Fourier correction. Together, these results support a contribution from temporal modeling beyond the spectral pathway alone.

2. Directional ablations

For the No Grid-to-Token ablation, independent runs including seeds 0, 1, and 2 yielded an NS2D relative L2L_2 error of 0.047±0.0028160.047 \pm 0.002816 (mean ±\pm standard deviation). The full-model result reported in Table 7 is 0.045.

The two interaction directions serve complementary roles. Grid-to-Token refreshes temporal tokens with current spatial information through lightweight convolutions and learned downsampling. Token-to-Grid applies token-conditioned spectral residual corrections to the grid. Because the tokens already receive spatial information at initialization, removing Grid-to-Token eliminates repeated spatial updates while retaining that initial spatial information, which may explain its relatively small impact on NS2D. Both directions use learned gated fusion to regulate the exchange between branches.

To characterize the learned fusion, we measured the mean weight assigned to the grid candidate in the Grid-to-Token pathway during autoregressive evaluation:

DatasetMean grid-candidate mixing weight
NS2D65.1%
SWE2D71.6%

These weights are averaged across samples, blocks, spatial positions, channels, and rollout steps, indicating that the trained model substantially incorporates updated grid information. In the reported directional ablations, removing Grid-to-Token increases relative L2L_2 error by 4.4% on NS2D and 19.9% on SWE2D, with a larger benefit on SWE2D.

We also profiled the computational costs of the two modules within the full model:

Cost metric per blockGrid-to-TokenToken-to-Grid
Parameters, including the fusion gate95,232733,568
Operations per sample per time step84.1 MFLOPs873.9 MFLOPs

Grid-to-Token requires approximately one-tenth of the operations and one-eighth of the parameters of Token-to-Grid. These results support complementary, asymmetric roles for the two directions: Token-to-Grid provides spectral correction, while the less expensive Grid-to-Token pathway supplies repeated spatial updates, with benefits that vary across tasks.

Overall, the two directions are not intended to contribute equally. Their asymmetric functional roles, learned gated information exchange, and complementary performance benefits justify the bidirectional architecture, with Grid-to-Token providing an especially lightweight and task-dependent spatial refinement pathway.

R2 - Q2

Q2: Requested evidence to resolve the training-protocol concern: Does GTNO retain its advantage over a competitive spatial or bottleneck-memory baseline when both use the same temporal training objective, with matched observed history, supervised target positions, and comparable training budgets? A comparison on one representative fluid benchmark would address this concern. How much of the training-time difference remains under that matched protocol?

A2: Thank you for highlighting the need to distinguish architectural gains from differences in the training protocol. We verified that the compared implementations aggregate the training loss over the same 10 supervised future frames. Validation is performed separately using a full 10-step autoregressive rollout.

To further examine the role of the temporal training formulation, we trained a serial GTNO variant following the conventional fixed-window setting. In this variant, the 10 historical frames are concatenated along the feature dimension, and temporal attention is disabled. Temporal information is therefore encoded in the input features, and the token branch no longer explicitly models temporal dependencies through causal attention.

The results on NS2D are summarized below:

ModelRelative L2L_2 error ↓Training time per epochPeak training memory
Full GTNO0.045356∼100\sim 100 s15.18 GiB
Serial GTNO0.086916∼203\sim 203 s18.33 GiB
Transolver0.87057∼192\sim 192 s15.03 GiB

The serial GTNO variant retains lower prediction error than Transolver in this comparison, although its per-epoch training time is slightly higher, at approximately 203 s versus 192 s. The full model achieves the lowest error and requires approximately 100 s per epoch, roughly half the time of the serial variant.

These results are consistent with the value of GTNO's explicit causal temporal modeling and parallel processing of temporal positions during training. However, because the serial variant also disables temporal attention, it changes both the temporal representation and the training formulation. This comparison therefore evaluates the full temporal modeling formulation and does not isolate the effect of the training schedule alone.

R2 - Q3

Q3: Long-horizon autoregressive forecasting. Table 4 evaluates NS2D and NS2D-PA over 10 predicted steps and CNS2D over 11, while DR2D and SWE2D include 91-step forecasts. Can GTNO maintain useful predictive accuracy over a few hundred autoregressive steps, for example 200–500 [1], on a Navier–Stokes benchmark with sufficiently long reference trajectories? Please keep the evaluation time interval fixed and report the corresponding physical duration. Per-step error or correlation, together with a spectral diagnostic, would help distinguish the useful prediction horizon from longer-term physical behavior. This would substantiate the long-horizon discussion in Section 4.3, particularly given the importance of low-amplitude spatial frequencies for long-rollout accuracy demonstrated by PDE-Refiner [2].

[1] Kochkov et al. "Machine learning–accelerated computational fluid dynamics." PNAS, 2021.

[2] Lippe et al. "PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers." NeurIPS, 2023.

A3: Thank you for this suggestion. We conducted an additional 300-step fully autoregressive evaluation on the 2D Kolmogorov flow benchmark at Re=1000Re=1000 and a spatial resolution of 64×6464 \times 64.

The evaluation uses 16 held-out trajectories, with five non-overlapping prediction windows per trajectory, yielding 80 evaluation windows. Both models begin with 10 ground-truth observations and predict 300 future steps without ground-truth feedback. GTNO uses an inference configuration consistent with its temporal training setup. The sampling interval is fixed at Δt=0.070124836\Delta t=0.070124836, so the 300-step rollout spans 21.04 simulation time units.

1. Per-step prediction accuracy and useful horizon.

We report relative L2L_2 error and vorticity correlation at individual forecast steps to characterize how trajectory accuracy changes over time. Durations are expressed in simulation time units.

Step $ H $DurationGTNO L2 ↓Transolver L2 ↓GTNO corr. ↑Transolver corr. ↑
100.700.00810.97630.99950.3967
503.510.07031.27470.98590.0133
1007.010.47411.26130.71230.0197
20014.021.35261.25560.07160.0235
30021.041.34941.25250.06200.0125

GTNO maintains high trajectory accuracy at step 50, with a relative L2L_2 error of 0.0703 and a vorticity correlation of 0.9859. By this point, Transolver has largely lost correlation with the reference trajectory. At step 100, GTNO still captures part of the trajectory structure, with a correlation of 0.7123, although its prediction error has increased.

Using 0.8 as a descriptive correlation threshold, GTNO first falls below this threshold at step 92, or 6.45 simulation time units, compared with step 6, or 0.42 simulation time units, for Transolver. This indicates a substantially longer useful prediction horizon in this benchmark. Nevertheless, both models are largely decorrelated from the reference trajectories by steps 200–300.

2. Cumulative long-horizon error

We also report prefix relative L2L_2 error, calculated jointly over the predicted fields from steps 1 through HH. This metric summarizes error across the entire rollout prefix and is distinct from the error at step HH.

Horizon HHGTNO prefix relative L2L_2 ↓Transolver prefix relative L2L_2 ↓Error reduction
500.03361.169897.1%
1000.18371.225585.0%
2000.77881.233736.9%
3001.01401.237218.0%

GTNO achieves lower cumulative error at every reported horizon. Since the prefix relative L2L_2 error is computed over the entire prediction horizon, it also incorporates the more accurate early predictions, resulting in a lower overall error than that measured at the final prediction step.

Reviewer_bhxMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Grid–Token Neural Operator (GTNO), a neural operator architecture for time-dependent PDE forecasting. The method maintains two complementary representations: a high-resolution spatial grid processed by ConvNeXt-style blocks and compact temporal tokens processed by causal self-attention. These representations are coupled through bidirectional interactions, where spatial features are aggregated into temporal tokens and temporal information is incorporated into the spatial grid through Fourier-based spectral corrections. The proposed design aims to preserve fine-grained spatial information while reducing the computational cost of temporal modeling. Experiments on five PDE benchmarks show that GTNO achieves lower forecasting errors than eight neural operator baselines. Additional ablation studies investigate the contributions of the two branches and their interactions.

Critical strengths
  1. Effective integration of spatial and temporal modeling. The architecture combines high-resolution spatial representations with compact temporal tokens, providing a reasonable approach to balancing spatial detail and temporal modeling efficiency.

  2. Comprehensive experimental analysis. The paper includes ablation studies, computational efficiency comparisons, and analyses of temporal ordering and spatial resolution, providing useful insights into the proposed architecture.

Critical weaknesses
  1. Limited novelty. The overall architecture appears closely related to existing neural operators that combine spatial processing with explicit temporal modeling, particularly MemNO. The spatial and temporal branches mainly rely on standard ConvNeXt blocks and causal attention, respectively. Although the proposed grid–token interaction introduces some architectural differences, I am not convinced that these modifications constitute a sufficiently substantial contribution. The authors should clarify what fundamentally distinguishes GTNO from existing spatial–temporal neural operators.

  2. Missing comparisons with closely related methods. Despite discussing MemNO and ENMA in the related work, the paper does not include them in the main benchmark comparisons. These methods seem more directly relevant to the proposed architecture than several of the selected baselines. Without such comparisons, it is difficult to assess whether GTNO provides meaningful improvements over existing approaches to temporal modeling.

  3. Insufficient evidence for the proposed bidirectional interaction. The ablation results in Table 7 raise some questions about the claimed importance of bidirectional coupling. On NS2D, removing the grid-to-token interaction barely changes the error (0.045 to 0.047), whereas removing the token-to-grid pathway substantially degrades performance. This makes me wonder whether the improvements mainly come from the spectral correction mechanism rather than the proposed bidirectional interaction. A more controlled ablation separating spectral correction from temporal conditioning would help clarify this point.

  4. Concerns about baseline fairness and computational efficiency. The baseline configurations are not fully comparable. For example, DPOT is trained from scratch despite being designed for large-scale pretraining, while SAOT uses a reduced architecture due to memory limitations. These choices raise concerns about the fairness of the comparisons. More importantly, although the paper emphasizes computational efficiency, Figure 6 in the appendix shows that GTNO has higher inference time and memory consumption than several baselines. Since fast inference is a key motivation for neural PDE solvers, I am not convinced that the claimed efficiency advantage is sufficiently supported by the experimental results.

Questions for authors
  1. What are the fundamental differences between GTNO and existing spatial–temporal neural operators, particularly MemNO?

  2. Could the authors provide a more direct comparison with MemNO and ENMA? These methods seem particularly relevant to the proposed architecture, and their absence makes it difficult to assess the significance of the reported improvements.

  3. In Table 7, removing the grid-to-token interaction has little effect on NS2D, whereas removing the token-to-grid pathway substantially degrades performance. Could the authors clarify whether the improvements mainly come from the spectral correction mechanism rather than bidirectional interaction? A controlled ablation would be helpful.

  4. How do the authors justify the claimed computational efficiency given the relatively high inference time and memory consumption shown in Appendix Figure 6? Could they also clarify how training DPOT from scratch and reducing the SAOT architecture affect the fairness of the baseline comparisons?

8 comments

R1 — Q1

Thank you for this question. While both GTNO and MemNO model spatial and temporal dependencies, they differ fundamentally in modeling responsibilities and state evolution.

AspectMemNO (S4FFNO)GTNO (Ours)
Modeling responsibilitiesSpatial and temporal modeling at the same spatial resolutionHigh-resolution grid for spatial modeling; low-resolution tokens for temporal modeling
State evolutionSequential composition of spatial and temporal layers (S–S–T–S–S)Persistent dual-resolution states that co-evolve through bidirectional gated interaction

Unlike MemNO's sequential spatial–temporal processing, GTNO maintains both representations throughout the network, allowing them to repeatedly exchange information through Grid-to-Token and Token-to-Grid interactions.

Thus, our key contributions and fundamental distinctions lie in assigning spatial and temporal modeling to different spatial resolutions and enabling their collaborative evolution through persistent bidirectional interaction.

R1 — Q2

Thank you for this suggestion. We have conducted additional comparisons with MemNO and ENMA.

MemNO: We did not include MemNO originally because neither official results under our benchmark settings nor an official code implementation was publicly available. We therefore independently reproduced its S4FFNO variant, following the main configurations described in the paper. We report full-rollout relative $ L_2 $ errors.

DatasetGTNO ↓MemNO ↓Error reduction
FNO NS2D0.0450.05173613.0%
DR2D0.0070.01684958.5%
CNS2D0.0980.12112819.1%
SWE2D ($ \times 10^{-3} $)0.6932.70974.4%
PDEArena NS2D0.2820.33578316.0%

GTNO achieves lower errors across all five benchmarks, with reductions ranging from 13.0% to 74.4% compared with our MemNO reproduction.

ENMA: ENMA is a flow-matching-based generative method requiring two-stage training and iterative sampling at inference. We evaluated its official implementation using the original 8/5 configuration, with 500 epochs of VAE pretraining and 1,000 epochs of forecaster training.

DatasetGTNO ↓ENMA ↓
FNO NS2D0.0450.232116

The ENMA result is averaged over generation seeds with fixed model weights. This comparison is provided for reference only, as the training procedures differ substantially, and a strictly unified evaluation protocol is difficult to establish due to ENMA's iterative generative inference and sampling-dependent performance.

We will incorporate these additional results and clarify the reproduction settings and evaluation differences in the revised manuscript.

R1 — Q3

Thank you for this observation. As detailed in our additional temporal ablation (R3–Q1) and fusion-gate analysis (R3–Q2), the performance gains cannot be attributed to spectral correction alone, but also involve temporal information and bidirectional interaction.

First, the performance gains of the Spectral Bridge cannot be explained by spectral correction alone. At the $ \text{training window} = 10 $, blocking cross-frame attention increases error by 123.67%, despite retaining Fourier correction. This indicates that temporal information plays an important role beyond spectral correction alone.

Second, the two interaction directions are inherently asymmetric in both their roles and computational costs. Token-to-Grid, which includes the Spectral Bridge, has a larger impact on performance, while Grid-to-Token provides lightweight spatial updates at approximately one-tenth of its computational cost. Although removing Grid-to-Token increases error by only 4.4% on NS2D, it causes a 19.9% increase on SWE2D, demonstrating task-dependent benefits. Moreover, the learned Grid-to-Token fusion weights (65.1% on NS2D and 71.6% on SWE2D) indicate that the model actively incorporates information from the grid branch.

Overall, the larger contribution of Token-to-Grid reflects the asymmetric design rather than the redundancy of bidirectional interaction. Spectral correction and temporal information both contribute to the Spectral Bridge, while Grid-to-Token provides complementary spatial updates with low computational overhead.

R1 — Q4

Thank you for raising these concerns. We clarify the computational efficiency of GTNO and the fairness of the baseline comparisons below.

1. Computational efficiency. GTNO is designed to enable parallel temporal processing during causal training, and our paper reports its favorable training efficiency. At inference, however, maintaining the KV cache and performing cross-frame attention introduce additional computational and memory overhead. For a fair comparison, GTNO retains a 10-frame KV cache, matching the historical context length of the fixed-window baselines.

We conducted unified profiling on a dedicated RTX 3090 using identical validation data and FP32 precision.

ModelNS2D Time (s)Mem. (MiB)SWE2D Time (s)Mem. (MiB)
GTNO17.44107092.12562
Transolver14.5320377.11130
SAOT11.7919773.26118

Times cover complete autoregressive evaluation (200 NS2D trajectories with 10 prediction steps; 100 SWE2D trajectories with 91 steps). We report median times over three runs and peak allocated GPU memory.

Although GTNO incurs higher inference costs, these are accompanied by improved prediction accuracy. For a fair accuracy comparison, we evaluated GTNO and SAOT using the same number of random seeds under a unified evaluation protocol. The reported errors are obtained from this additional seed-count-matched evaluation and may therefore differ slightly from those in the submitted manuscript. Under this setting, GTNO reduces relative $ L_2 $ error by 38.6% on NS2D ($ 0.07219 \rightarrow 0.04433 )and∗∗24.4) and **24.4% on SWE2D** ( 0.000870 \rightarrow 0.000658 $) compared with SAOT.

Importantly, GTNO supports adjustable inference windows without retraining, allowing users to balance accuracy and resource consumption. We investigate this flexibility through the inference-window analysis presented in Figure 7. Unlike models requiring fixed-length histories, GTNO can also start prediction from a single frame without padding. In a separate NS2D window-sensitivity evaluation with matched seed counts, reducing the inference window from 10 to 7 decreases peak allocated memory from 1070 to 853 MiB, with only a small increase in error from 0.04433 to 0.04586.

Overall, we believe that the architectural advantages of parallel temporal processing during training, flexible inference windows, and high predictive accuracy provide a favorable trade-off against the additional inference time and memory costs.

2. DPOT comparison. We acknowledge that DPOT was primarily designed for large-scale PDE pretraining. However, the original paper also explicitly evaluates DPOT-Vanilla trained from scratch for 500 epochs (Section 4.4, Table 2), which outperforms the (Geo-)FNO baseline across all three downstream tasks. This demonstrates that the DPOT architecture remains competitive without pretraining. Therefore, we use DPOT trained from scratch to compare architectural performance under the same target-data-only setting. We will clarify this scope in the revised manuscript.

3. SAOT comparison. The submitted SAOT baseline uses a reduced configuration due to GPU memory constraints. To address the fairness concern, we additionally evaluated the full 8-layer, width-256 SAOT on FNO NS2D with batch size 2, consistent with its original configuration. Batch size 4 causes out-of-memory errors on our 24-GB RTX 3090.

Model / ConfigurationNS2D $ L_2 $ ↓Training time / epoch
GTNO0.045$ \sim 100 $ s
SAOT (6 layers, width 192)0.072$ \sim 320 $ s
SAOT (8 layers, width 256)0.0708$ \sim 410 $ s

Restoring the full SAOT architecture reduces its error by only 1.7%, while increasing the observed per-epoch training time by approximately 28%. GTNO still achieves 36.4% lower error than full SAOT, with substantially shorter training epochs under their respective training configurations. We will report both SAOT configurations and their training settings explicitly.

Reviewer 1

Additional Results of (R1–Q2): Comprehensive Comparisons with MemNO and ENMA. (Could the authors provide a more direct comparison with MemNO and ENMA?)

A1: Following our previous response to R1–Q2, we have now completed additional ENMA experiments, extending the comparison with MemNO and ENMA to all five benchmarks.

Full-rollout relative L2L_2 error (↓ lower is better). The best result on each dataset is highlighted in bold.

Model / DatasetFNO NS2DDR2DCNS2DSWE2DPDEArena NS2D
GTNO0.0450.0070.0980.0006930.282
MemNO0.0517360.0168490.1211280.0027090.335783
ENMA0.2321160.9754540.5616670.0597130.570410

As shown, GTNO consistently achieves the lowest prediction errors across all five benchmarks, outperforming both MemNO and ENMA. These additional results further support the effectiveness of GTNO. We will incorporate the complete comparisons into the revised manuscript.

We would also like to kindly follow up as the discussion period is approaching its deadline. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. We would be more than happy to provide any further clarifications or engage in additional discussion if needed.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.