acceptodds
Withdrawn from ICLR 2027

Symbol-conditioned Adaptive Fourier Operator Transformer for Large-Scale PDE Pre-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural operators have emerged as a promising paradigm for solving PDEs and building PDE foundation models. However, existing architectures often suffer from spectral bias and struggle to capture high-frequency dynamics, limiting their capacity to model complex physical systems. To address these challenges, we introduce the Symbol-conditioned Adaptive Fourier Pre-training Operator Transformer (SAFPOT). First, we define multi-level frequency bands to represent heterogeneous spectral regions, enabling full-frequency learning. First, the multi-level frequency bands are defined to represent heterogeneous spectral regions, enabling full-frequency learning. Second, we design a novel Symbol-conditioned Adaptive Fourier (SAF) module, which dynamically modulates low-, mid-, and high-frequency bands via a frequency gate network. Together, these designs allow the model to adapt its spectral modeling strategy to the characteristics of each governing equation and effectively capture multi-scale physical dynamics. Finally, we pre-train models with parameters from 40M to 0.5B on 6 PDE datasets. In data-oriented evaluation, pre-trained SAFPOT with only 28M activated parameters surpasses fully fine-tuned baselines and further reduces L2RE by up to 71.75% after PDE-specific fine-tuning. In physics-oriented evaluation, SAFPOT reduces the average fRMSE by up to 59.4% compared with the strongest baseline and substantially reduces errors across each frequency band.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Neural Operators, Scientific Machine Learning, PDE Foundation Models, Operator Learning, Fourier Neural Operators

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iyULOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SAFPOT, a unified pretraining framework for PDE solution prediction that conditions frequency-band modulation on both physical states and symbolic PDE representations. It combines spectral feature modulation with mixture-of-experts (MoE) layers to capture equation-specific dynamics. Experiments on pretraining, transfer learning, ablations, and model scaling show promising improvements, with reported reductions of up to 71.75% in L2RE and 59.4% in average fRMSE.

Critical strengths

Different PDEs exhibit distinct spectral distributions. Jointly conditioning frequency-band modulation on PDE symbols and the current physical state provides a targeted architectural design for multi-PDE operator learning.

Critical weaknesses
  1. The sources and comparability of baseline results are unclear, raising concerns about the reported performance gains. Several errors in Table 1 differ substantially from those in the original papers, including MoE-POT on SWE, CNS, and DR, as well as NESTOR under the same fine-tuning epochs. These discrepancies are difficult to assess due to insufficient details on data splits, implementations, and evaluation protocols. Notably, the claimed maximum error reduction of 71.75% depends on the elevated SWE baseline error. The authors should clarify the sources of baseline results and verify the gains under a consistent experimental protocol.
  2. The lack of model checkpoints and incomplete evaluation details hinder independent verification and reproducibility. The public repository does not provide checkpoints for the main results, while key details regarding dataset splits, prediction horizons, error aggregation, FFT normalization, and frequency-band boundaries remain unclear. These omissions, combined with the baseline discrepancies, weaken confidence in the empirical conclusions. The authors should provide the relevant checkpoints, evaluation commands, and complete configurations.
Questions for authors
  1. Why is MoE-POT-T’s SWE error substantially higher than that reported in the original paper?

Both papers use six pretraining datasets and 200 fine-tuning epochs, yet the reported SWE errors are 0.00945 and 0.00305, respectively. What differences in training or evaluation explain this discrepancy? It directly affects the claimed maximum error reduction of 71.75%.

  1. Why do some baselines degrade after fine-tuning, and why do their results differ from those in the original papers?

In Table 1, NESTOR’s average error increases from 0.0205 before fine-tuning to 0.0225 afterward, while MoE-POT-T’s SWE error increases from 0.00405 to 0.00945.

Furthermore, after 200 fine-tuning epochs, the reported errors differ from the original results for NESTOR on FNO-ν ((10^{-5})), DR, and SWE (0.0761, 0.0266, and 0.00366 vs. 0.0581, 0.0198, and 0.0032), as well as for DPOT-T, DPOT-S, and DPOT-M on SWE (0.00394, 0.00560, and 0.00264 vs. 0.00280, 0.00224, and 0.00218). How do the authors explain these discrepancies? Which are attributable to differences in pretraining data versus training or evaluation settings? Please clarify the fine-tuning procedures and checkpoint-selection criteria used for each baseline.

  1. Does the symbolic branch capture governing-equation information or primarily identify the dataset?

The current ablation compares SAFPOT with and without the symbolic branch but does not distinguish these explanations. Would replacing the equation symbols with learnable dataset identifiers, while keeping the frequency-modulation structure unchanged, preserve SAFPOT’s performance advantage? This distinction is central to evaluating the contribution of symbolic conditioning.

Reviewer_7YwNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SAFPOT combines historical physical fields with symbolic PDE representations for neural-operator pre-training. Its SAF module generates state-dependent spectral responses and applies symbol-conditioned modulation across three soft frequency bands. The model uses an MoE architecture with prediction, alignment, and load-balancing losses. Experiments cover six pre-training datasets, three model sizes, and three downstream tasks.

Critical strengths
  1. Clear motivation and a concrete mechanism. Conditioning spectral processing on both physical states and equations is a reasonable response to differing PDE spectra. Eqs. (7)–(10) clearly describe how symbolic information modulates the spectral response.

  2. Broad predictive improvements. In Table 1, SAFPOT-M improves average L2RE over MoE-POT-M from 0.0202 to 0.0156 before fine-tuning and from 0.0131 to 0.0102 afterward. It also achieves the best reported results on all three downstream tasks in Table 4.

Critical weaknesses
  1. Novelty and the specific value of equation semantics remain insufficiently established. PROSE-PDE and PROSE-FD already encode symbolic equations and fuse them with physical data, but neither is discussed. Given the adoption of MoE-POT’s expert architecture, the contribution should be distinguished more clearly through the proposed spectral modulation mechanism.

Table 3 shows only a 3.8% average improvement from adding the symbol branch (0.0184 → 0.0177), with worse results on CNS and CFDBench. Run-to-run variability, ablation parameter counts, and band-wise ablation results are not reported. Moreover, dataset-associated symbolic inputs and same-PDE alignment positives leave open whether symbols primarily serve as task identifiers rather than conveying equation structure or coefficient semantics.

  1. Frequency-domain evaluation and result presentation need clarification. Taken literally, Eq. (14) averages error magnitudes rather than explicitly computing an RMSE. The accompanying description leaves the 3D FFT axes, normalization, radial grouping, and time/channel aggregation unclear.

Fig. 4’s right panel is labeled NS (10⁻⁵), but its SAFPOT values appear to match Table 1’s cross-dataset averages, unlike the dataset-specific values in the left panel. The abstract also needs to qualify its comparison: the pre-trained 28M model outperforms the fine-tuned Tiny baselines on average, but not the larger fine-tuned models.

Reference:

[1] Sun, J., Liu, Y., Zhang, Z., and Schaeffer, H. Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation. arXiv:2404.12355, 2024.

[2] Liu, Y., Sun, J., He, X., Pinney, G., Zhang, Z., and Schaeffer, H. PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics. arXiv:2409.09811, 2024.

[3] Hao, Z., et al. DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training. arXiv:2403.03542, 2024.

Questions for authors
  1. What does the symbol branch add beyond task identity?

How are parameter budgets and the alignment loss handled in the no-symbol ablation? Are task-ID controls or multi-seed results available? PROSE-FD’s approximately parameter-matched data-only comparison provides a useful reference.

  1. What distinguishes the proposed mechanism from prior fusion methods?

Please discuss PROSE-PDE and PROSE-FD. DPOT also applies a nonlinear MLP to input spectra, so please distinguish fixed learned parameters from input-dependent spectral responses. Can existing ablations isolate the explicit multiplicative response and symbol-conditioned modulation?

  1. How is fRMSE implemented?

Please specify FFT axes and normalization, squaring/averaging order, frequency bands, and time/channel aggregation. Do all baselines use the same evaluator, and do band-wise symbol ablations support the claimed high-frequency benefit.

  1. Could you resolve the figure and implementation ambiguities?

Please clarify Fig. 4’s right-panel data and specify symbol-sequence aggregation, frequency thresholds, soft-mask temperature, modulation magnitude, and loss weights.

Reviewer_k2zmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SAFPOT, a multi-PDE pre-training architecture combining state-dependent spectral transformations, symbolic equation conditioning, and mixture-of-experts layers. PDE expressions are serialized into token sequences and encoded to modulate low-, mid-, and high-frequency bands jointly with the current state features. Training combines prediction, cross-modal alignment, and expert load-balancing objectives. The authors evaluate three model sizes on six pre-training datasets, followed by dataset-specific fine-tuning and transfer to three downstream tasks. The reported results show improvements in average relative prediction error and frequency-domain error, with component ablations examining frequency modulation, symbolic inputs, and alignment.

Critical strengths
  1. Broad empirical evaluation with promising results. The paper evaluates multiple model scales, reports both spatial and frequency-domain errors, and includes downstream comparisons between training from scratch and fine-tuning. For example, the Medium model reduces average fine-tuned L2RE from 0.0131 for MoE-POT-M to 0.0102, with similar reported activated parameter counts.

  2. A motivated approach to heterogeneous PDEs. Using equation information to condition spectral processing is an interesting design choice. The ablations support the usefulness of frequency modulation and show a smaller average improvement from the symbolic branch.

Critical weaknesses
  1. The relationship to existing equation-conditioned models needs clearer discussion. Closely related approaches such as PROSE-FD [1] and PDEformer-2 [2] are absent from the discussion and comparisons. Their equation representations and conditioning mechanisms provide important context for assessing the specific contribution of frequency-band modulation. A suitable equation-conditioned baseline would strengthen the evaluation, while accounting for differences in task formulation and pre-training resources. Recent equation-conditioned amortized PINN work [3] also provides a useful conceptual connection, although its physics-based training and adaptation setting differs.

  2. The contribution of equation structure is not yet isolated. Removing the symbol branch increases average L2RE from 0.0177 to 0.0184. This supports its usefulness, but does not distinguish structured equation information from simpler task-conditioning information. A dataset-ID or coefficient-only conditioning baseline would be informative, particularly because the alignment objective groups samples by PDE. The downstream experiments evaluate the complete model, so they do not establish how much of the transfer benefit comes from symbolic conditioning. Concrete symbolic-input examples and a clearer description of sequence pooling into the frequency gate would also help interpret this mechanism.

  3. Some baseline behavior and comparison terminology warrant clarification. In Table 1, fine-tuning worsens several baseline results: NESTOR’s average L2RE increases from 0.0205 to 0.0225, while MoE-POT-T’s SWE error increases from 0.00405 to 0.00945. The latter is the reference for the highlighted 71.75% reduction. The percentage is arithmetically correct, but its interpretation would benefit from explaining this deterioration and the validation/model-selection procedure. Additionally, the “scale-matched” Tiny comparison uses 28M activated parameters for SAFPOT versus 17M for MoE-POT and 7.5M for DPOT. These observations do not negate the reported gains, but suggest that more precise budget comparisons and fine-tuning details are needed.

References

[1] PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics. https://arxiv.org/abs/2409.09811

[2] PDEformer-2: A Versatile Foundation Model for Two-Dimensional Partial Differential Equations. https://arxiv.org/abs/2507.15409

[3] Amortizing Physics-Informed Neural Solvers via Graph Hypernetworks. https://arxiv.org/abs/2609.19915

Questions for authors
  1. Could you compare against a suitable equation-conditioned model, such as PROSE-FD or PDEformer-2, and clarify the differences in conditioning mechanisms and experimental settings?

  2. Could you compare symbolic conditioning with dataset-ID and coefficient-only conditioning, including on downstream tasks, and report variability across seeds?

  3. Could you investigate the baseline degradation after fine-tuning and report validation-selected results under comparable tuning and computational budgets?

  4. Could you provide the symbolic-input examples, encoder/gating configurations, and training/evaluation details needed to reproduce these comparisons?

  5. Could you add a related-work subsection on equation-conditioned solvers, clarifying their connections to and differences from the proposed approach?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.