acceptodds
Under review as a conference paper at ICLR 2027

SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining

Abstract

Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still face heterogeneous dynamics, where shared representations may cause knowledge interference and MoE architectures may introduce expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR separates latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To reduce redundancy, we further introduce a knowledge-guided aggregation strategy that measures expert similarity using dataset-specific knowledge and routing preferences, and consolidates similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate strong performance in pre-training, fine-tuning, and transfer learning. Our aggregation strategy further reduces the number of experts by 50% while maintaining or improving prediction accuracy, balancing efficiency and generalization.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neural Operator, PDE Foundation Model, Mixture of Experts, Expert Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_WRy3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPEAR is a mixture-of-experts neural operator for pretraining across heterogeneous PDE datasets. An AFNO branch is meant to model shared low-frequency dynamics, and a local convolutional MoE branch with expert-specific low-rank adapters (top-2 routing) is meant to capture PDE-specific high-frequency structure. A knowledge-guided aggregation step projects dataset similarities and dataset–expert routing preferences into expert space, clusters the experts and merges their adapters. Experiments cover twelve PDE datasets, task-specific fine-tuning, transfer to turbulence and wave benchmarks, and component ablations.

Critical strengths
  1. Combining a shared AFNO backbone with lightweight routed adapters is a sensible way to share capacity across PDE tasks while keeping task-specific behaviour, and projecting dataset similarity into expert space is an interesting idea for organizing experts.
  2. The evaluation is broad: twelve datasets, downstream transfer, expert-count variations and aggregation variants. Removing the additional branch clearly hurts (Table 5), and fewer routing destinations retain accuracy after fine-tuning (Table 1).
Critical weaknesses
  1. The supplied code does not implement the claimed spectral split. With the settings in Table 8 (128×128 input, patch 8, 32 modes) the latent grid is 16×16 and its rFFT is 16×9, so AFNO2D.forward, which slices [:32, :32], processes the entire spectrum, and _fft_high_mask zeroes only the leading 9×9 block, which is not a magnitude high-pass because the first axis holds both positive and negative frequencies (cf. Eqs. 3–4). The ablation in Table 5 thus shows the value of an extra branch, not of spectral disentanglement, and the relation to FreqMoE, which already combines low-frequency processing, high-frequency experts and low-rank adaptation, is not discussed.
  2. The aggregation and compression claims are not reproducible from the materials. The supplied checkpoint routes to six destinations per layer but still stores twelve adapter pairs per layer (about 1.57M parameters over four layers, the pre-merge figure in Table 7, not the 0.786M reported after merging), and the available merge_similar_experts averages adapters by weight similarity rather than implementing Eqs. 9–12; the similarity and routing matrices, reused from NESTOR (Appendix A.6), are not provided. Merged-model accuracy is reported only after further fine-tuning, so the loss caused by merging itself is not measured.
Questions for authors
  1. (Clarification) Which implementation produced the reported results, and what are the effective spectral cutoff and latent resolution? Do the main results hold with a frequency-consistent mask, and against a parameter-matched local branch without filtering?
  2. (Clarification) Can you release the dataset-similarity and routing matrices, their construction, and the code that produced the merging groups, and report accuracy immediately after merging, before fine-tuning?
  3. (Clarification) After physically removing merged experts, what are the stored and activated parameter counts, checkpoint size under identical serialization, memory and latency?
  4. (Clarification) Were all baselines in Table 3, in particular Poseidon, re-evaluated under the same data size, horizon, fine-tuning budget and L2RE metric? Poseidon's original evaluation uses final-time relative L1 error.
Reviewer_VzwYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPEAR, a spectral-disentangled MoE neural operator for large-scale PDE pre-training. The method aims to address two challenges: knowledge interference across heterogeneous PDE systems and parameter redundancy among MoE experts. SPEAR introduces a dual-branch architecture that uses a shared AFNO module for low-frequency representations and a LoRA-based MoE module for high-frequency representations. It further proposes a knowledge-guided expert aggregation strategy that combines dataset-specific fine-tuning information with expert routing preferences to identify and merge redundant experts.

Critical strengths
  1. The paper addresses two meaningful challenges in large-scale PDE pre-training: knowledge interference across heterogeneous PDEs and expert redundancy in MoE-based neural operators.
  2. The proposed architecture combines frequency decomposition with shared and specialized experts. The overall design is relatively simple and intuitive, and the use of LoRA adapters helps control the number of model parameters.
  3. The experiments cover 12 PDE datasets and evaluate pre-training, fine-tuning, and transfer learning. The proposed method achieves competitive performance against several neural operator and PDE foundation model baselines.
Critical weaknesses
  1. There appears to be an inconsistency between the frequency decomposition equations and the experimental configuration. According to Appendix A.1, patch embedding reduces the spatial resolution from 128 × 128 to 16 × 16 when the patch size is 8. However, Table 8 specifies 32 Fourier modes. If this corresponds to the cutoff K in Eq. (4), the high-pass mask would remove all spatial frequency components, resulting in an all-zero input to the high-frequency branch. The authors should clarify how the Fourier modes and high-pass filtering are implemented.
  2. The practical efficiency improvements are not sufficiently demonstrated. Although the number of non-shared experts is reduced from 12 to 6, Table 7 shows that only approximately 0.787M expert parameters are removed. Moreover, the reported checkpoint size reduction from 138 MB to 47 MB appears difficult to explain solely by this parameter reduction. The authors should clarify how checkpoint sizes are measured.
Questions for authors
  1. In Eq. (12), the LoRA factors AiA_i and BiB_i are averaged independently. However, this operation does not generally preserve the weighted average of the effective parameter updates AiBiA_iB_i. Have the authors considered directly averaging AiBiA_iB_i and applying truncated SVD to recover low-rank adapters?
  2. Does the proposed method require independently fine-tuned models for all PDE datasets to construct the dataset similarity matrix? What is the total additional computational cost when such models are not already available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.