acceptodds
Withdrawn from ICLR 2027

Spatiotemporal-Aware Mamba Neural Operator for Time-dependent PDE Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural operators (NOs) have emerged as efficient surrogates for predicting PDE-governed spatiotemporal dynamical systems. However, accurately modeling both temporal evolution and spatial interactions remains challenging, particularly for turbulent and long-horizon dynamics. We introduce MCANO, an autoregressive Mamba-based neural operator with dedicated temporal and spatial processing. Specifically, the temporal-awareness module employs a lightweight pointwise polynomial-basis lifting and embedding strategy to explicitly characterize temporal evolution patterns at each spatial location, while the spatial-awareness module augments conventional directional scans with a learned offset-based scan to provide an additional feature-conditioned spatial traversal. Building upon MCANO, we further develop MCANOP, a scalable multiphysics pretraining and fine-tuning framework that combines random sub-trajectory sampling, instance-wise normalization, and task-specific adaptation for heterogeneous PDE systems. Across regular-grid, irregular-grid, short-trajectory, and long-trajectory benchmarks, MCANO achieves the best performance on six of nine datasets and ranks second on the remaining three. After task-specific fine-tuning, MCANOP-17M substantially outperforms pretrained models of comparable size across all in-domain and out-of-domain datasets. Our findings highlight the potential of MCANO and MCANOP as parameter-efficient paradigms for accurate PDE prediction, matching or surpassing the performance of significantly larger models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, Mamba, multiphysics pretraining, spatiotemporal modeling, partial differential equations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pacdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MCANO, a Mamba-based neural operator for time-dependent PDE prediction. The architecture separates pointwise temporal feature construction from spatial modeling: TAM combines polynomial temporal lifting with an MLP, while SA-Mamba integrates fixed and content-adaptive scans to process spatial features. The authors further introduce MCANOP, a framework for multi-PDE pretraining and task-specific fine-tuning. MCANO is evaluated on nine PDE datasets, including irregular-domain and long-horizon settings, with ablation studies assessing the contributions of its core modules. The reported results show competitive prediction accuracy and substantial benefits from pretraining followed by fine-tuning, including on downstream PDE datasets excluded from pretraining.

Critical strengths
  • Generalizability of the pretraining–fine-tuning paradigm: Pretraining followed by task-specific fine-tuning considerably improves prediction accuracy over training from scratch. The model also achieves competitive results after fine-tuning on downstream PDE datasets excluded from pretraining. These findings support its potential utility across a range of prediction tasks.
Critical weaknesses
  • Unclear motivation and research gap: The paper does not clearly articulate the specific limitations of existing methods, including prior Mamba-based PDE solvers. Accurately modeling spatiotemporal dependencies is a broad objective shared by essentially all approaches to time-dependent PDEs. This framing leaves the manuscript’s motivation and the concrete problem addressed by the proposed method insufficiently clear.

  • Insufficient validation of the proposed design mechanisms: For example, the adaptive scanning module is intended to make information propagation better reflect the spatial organization of the physical field. Although the ablation results demonstrate improved prediction accuracy, they do not establish whether the module achieves this intended behavior. The paper lacks visualizations of the learned sampling locations and scan paths on representative physical fields.

Questions for authors
  • Translation and rotation equivariance: When a physical scenario is translated or rotated, do the model’s predictions undergo the corresponding transformation? Could the authors clarify how the spatial encoding and scanning schemes affect these properties?

  • Scalability to 3D: How would the fixed and adaptive scanning schemes extend to three-dimensional domains, and would their computational and memory costs remain practical as volumetric resolution increases?

  • Transfer across geometries and boundary conditions: Can the pretrained model transfer to domain geometries or boundary conditions absent from pretraining? For example, can pretraining on simple geometries facilitate adaptation to more complex domains formed by combining basic shapes?

I consider the paper technically complete overall, with my main concerns being the unclear motivation and insufficient validation of the proposed modules’ design rationale, as discussed in the weaknesses. I would raise my score if the authors clearly articulate the motivation in the abstract and introduction and provide evidence that the modules achieve their stated design objectives.

Reviewer_us5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new neural operator based on Mamba for time-dependent partial differential equations. The problem is not new. The key technical contributions are a new model architecture that has spatial and temporal correlations modeled. Temporally, a specific module is introduced to embed time information with the solution itself; spatially, the order of scan is adjusted based on learned spatial and path offset. Experiments were done on popular benchmark datasets, with comparisons with small and large neural operators. The improved prediction accuracy appears promising.

Critical strengths

A wide range of comparisoin baselines and datasets. Improved accuracy in some cases.

Critical weaknesses

I have several major concerns of this paper.

  1. Understanding of literature and novelty positioning. There are comments that need to be clarified, as they serve as key anchors for positioning this paper.

In introduction, 'On the temporal side, historical states are commonly stacked as input channels and subsequently mixed by spatial operator blocks, providing limited explicit structure for describing temporal evolution.' Stacking historical states and subsequently mixing them does not automatically lead to limited explicit structure. The order and scan path are adjusted and feature-based in the propose method, so the propose method does not have explicit structure for describing temporal evoluation either. In addition, 'On the spatial side, Mamba-based models typically serialize two-dimensional features using predefined scan orders.' Predefined scan orders do not automatically lead to sub-optimal performance. Many Mamba models use predefined orders because the predefine order nicely described the data. Also, the opposite of predefined scan orders does not automatically mean better performance either. There is no emprical or theoretical proofs that these are truly limitations of existing research. In other words, the description here is simply saying the proposed method is different, not necessarily novel.

Next, 'Despite these advances, most existing Mamba-based NOs either treat time as an extra channel, missing explicit temporal dynamics, or apply fixed scanning paths that ignore the spatially varying structure of physical fields. ' have similar issues. Using time as an extra channel and using fixed scanning paths do not automatically mean missing temporal dynamics, or ignoring spatially varying structures. The papers cited by the paper actually show spatial and temporal dynamics of their predictions.

  1. Justification of methodologies. Following the above observations, the key technical solutions need more justifications.

In Eq. 4, it was proposed to use polynomial basis for encoding normalized time. This could impose some sequence information but why is it fundamentally different other alternative solutions, such as adding time as an extra channel. Furthermore, it is unclear why adding such an embeding increases the representation power because it is a fixed affine transformation on u. It looks like this is something a MLP can learn. This brings the question of the following use of MLP in Eq. 5. Is it actually the MLP is doing the actual work, not the lifting? Also, if the time information is so crucial, how do we know the MLP (Eq.5) will not destroy it?

Multiple scan paths is technically not new (see [1]). Learning the path is interesting, but it is searching a solution in a combinatorial space and it is unclear how the f_{offset} in Eq 6 does it. This part is particularly difficult to understand. What are p_i^0, B, and b_i^{rpe} in Eq. 7? Also, what does it mean 'where r_i denotes the fixed monotonic reference score of the raster order'? Is there a default or pre-defined raster order? In this case, doesn't \delta_i^{path} actually depend on the raster order, and so does q?

Finally, if a Convolutional Feed-Forward Network is used, how do we know it is not only learning local correlations?

  1. Crude evaluation.

The evaluation largely uses the metrics that are popular in prediction. However, they do not suffice for the claimed technical novelty. The improved accuracy is aggregate accuracy, not direclty attributing improvements to the proposed spatial and temporal modeling. This is especially so when, although explicit spatial and temporal structured are learned and imposed, it is unclear why the explicitly embeded time, spatial, and path orders led to improvements. More detailed analysis is needed to show it.

The above argument also applies to the ablation study. Blindly adding one module and improving results does not necessarily explain the effectiveness of the contribution of the proposed modeling techniques. It could be simply because of the increase of model capacity.

Baseline methods. Since multiple-PDEs are involved, models such as Poseidon and Walrus could be included.

Metrics on prediction accuracy are reasonable. However, physical fidelity metrics such as conservation of mass when possible should be included too.

  1. Writing. There are writing issues but they are likely to be easily fixable. There are also empty references such as line 1456 'As shown in Table ??'

[1] Latent Mamba Operator for Partial Differential Equations

Questions for authors

Please see my comment above.

Reviewer_oykEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MCANO, a Mamba-based model for predicting time-dependent PDEs. The model first encodes the history at each spatial location using polynomial-weighted features and a pointwise MLP. It then processes spatial interactions through four fixed scan directions and an additional scan that adjusts its sampling locations and traversal order based on the features. The authors also introduce MCANOP, which combines multi-PDE pretraining with task-specific fine-tuning. Experiments cover regular grids, irregular geometries, and different prediction horizons. MCANO achieves the best reported results on six of nine benchmarks, while the pretrained version shows further improvements after fine-tuning.

Critical strengths
  1. Competitive results across several PDE settings. Table 1 and Figure 3 report the best results on six of nine datasets and the second-best results on the remaining three, covering regular grids, irregular geometries, and different trajectory lengths. On NS with viscosity 1e-5, MCANO achieves an L2RE of 0.0459 versus 0.0658 for LaMO-L, an approximately 30% relative reduction. This is a meaningful empirical improvement and supports the practical promise of the architecture.

  2. The paper includes useful component and pretraining controls. Table 5 shows that the complete model outperforms the variants removing TAM or the dynamic scanning component across five datasets. Table 4 also compares the same 17M architecture under different training schemes: on NS with viscosity 1e-5, the error decreases from 0.0602 when trained from scratch to 0.0258 after pretraining and fine-tuning. These results provide evidence that the complete design and pretraining procedure are useful under the reported settings.

Critical weaknesses
  1. The contribution and learning mechanism of the principal modules are not sufficiently established. For fixed observation times and an unconstrained affine first MLP layer, TAM's lifting in Eqs. (4)–(5) is a fixed linear map, z = L_K u, which can be absorbed into that layer: Wz = (W L_K)u. The authors acknowledge that TAM is a reparameterization and provide a naive MLP comparison, but Appendix F's polynomial approximation result does not establish the claimed preference for smooth temporal dynamics. Optimization and implicit regularization remain plausible explanations for the improvement. For SA-SCAN, Eq. (8) uses hard argsort, yet the manuscript does not specify how the path-offset output receives gradients through the resulting permutation. This concern specifically concerns learning the ordering; the spatial sampling branch can remain differentiable. Furthermore, the combination of spatial-offset sampling and token-index offsets followed by sorting is closely related to DefMamba [1], whose relationship to the proposed method is not discussed. These gaps affect the interpretation and originality of the two central architectural contributions.

  2. The pretraining and transfer results lack a sufficiently clear and consistent evaluation protocol. Table 2 reports MCANOP's OOD results after fine-tuning and leaves its pretrained-only OOD entries blank. Appendix G nevertheless describes “zero-shot” OOD experiments, then reports 80 training epochs on PDEArena and an error of 0.0372, compared with 0.0125 in Table 2. Different checkpoints or settings could explain these numbers, but their correspondence is not provided. The DPOT comparison also needs clarification: Section 4.3 specifies 64×64 pretraining for MCANOP, whereas DPOT [2, Section 3.3] describes preprocessing at 128×128, and several DPOT entries match its published results. Different pretraining resolutions are not inherently problematic, but the manuscript does not establish a common evaluation grid, target variables, rollout horizon, and error aggregation, or identify which baseline results were quoted versus re-evaluated. Consequently, the headline accuracy and transfer advantages cannot yet be assessed under clearly comparable prediction tasks.

References: [1] Liu et al. DefMamba: Deformable Visual State Space Model. CVPR, 2025.

[2] Hao et al. DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training. ICML, 2024.

Questions for authors
  1. What backward rule is used for the path-offset prediction in Eq. (8)? Please provide the relevant implementation and explain how parameters specific to this output receive gradients from the prediction loss. Please also clarify the technical differences between SA-SCAN and DefMamba's spatial-offset and index-offset scanning mechanism.

  2. Given that TAM's fixed-grid lifting can be absorbed into an affine MLP input layer, what evidence supports the claimed smooth-temporal-dynamics bias beyond a benefit from reparameterization? Please specify the polynomial degree and MLP dimensions used in Table 3, and clarify whether the naive MLP comparison controls for hidden width, initialization, and training settings.

  3. Were the DPOT results in Tables 2, 24, and 25 quoted from the original paper or re-evaluated? Please specify the test splits, evaluation grids, predicted variables, rollout horizons, and L2RE aggregation used for both DPOT and MCANOP, including any resampling applied before computing the errors.

  4. Which OOD results involve no target-task training? Please reconcile the zero-shot description in Appendix G with its reference to 80 training epochs on PDEArena, and explain how the reported error of 0.0372 relates to 0.0125 in Table 2. Please identify the corresponding training settings and checkpoints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.