acceptodds
Under review as a conference paper at ICLR 2027

Time-Aware Operator Learning with Multi-Fidelity Solver-Embedded Mixture of Experts

Abstract

In numerically solving PDEs, achieving both generalization and efficiency is crucial but challenging. While recent advancements in neural operators provide significant speed-ups, they often struggle to generalize to out-of-distribution scenarios. Hybrid models that integrate neural networks with conventional numerical solvers offer improved generalization ability, but they incur high computational costs. To address this efficiency-generalization trade-off, we propose the Multi-Fidelity Mixture-of-Experts (MF-MoE) framework. MF-MoE dynamically allocates resources based on physical parameters and an optimal routing solver across expert models of varying fidelity, optimizing both computational cost and predictive accuracy. This innovative design enables faster inference for in-distribution inputs while ensuring better generalization for out-of-distribution cases. Extensive experiments on fluid flow prediction governed by the incompressible Navier-Stokes equations demonstrate that MF-MoE consistently outperforms baseline approaches, offering an efficient solution for PDE surrogate modeling.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Operator Learning, Numerical Solvers, Neural PDE, Navier–Stokes Equations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_J26hOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes the MF-MoE framework for PDE surrogate modeling. It combines one pure neural operator with three solver-based hybrid experts that run PhiFlow at Coarse, Medium, or Fine resolutions, upsample the result and blend it with a neural network output. To balance inference time and accuracy, the physical parameter space (viscosity/buoyancy) is discretized into bins. During training, a look-up table of historical MSE and wall-clock runtime per expert is maintained for each bin. At inference time, a small Linear Program computes a probability distribution over the experts to minimize expected MSE subject to an expected time budget. Experiments on 2D Navier-Stokes and 2D Burgers equations report lower test MSE than standalone neural operators while meeting a 10s average time constraint.

Critical strengths

Balancing the fast inference of pure neural operators against the ability of out-of-distribution generalization of classical numerical solvers addresses a real problem in scientific machine learning.

Critical weaknesses
  1. Invalid selector and gating The gating mechanism is a static per-bin look-up that cannot handle OOD inputs, which invalidates the paper's core claim. Despite claims of instance-wise routing that learns to determine whether using a physical solver is necessary for a given input, the gate never inspects the input physical field. It is a static table look-up indexed solely by the scalar physical parameter bin, followed by random sampling. Because the LP optimizes a linear objective over a 4-element simplex with one linear budget constraint, any basic optimal solution trivially places all probability mass on at most two adjacent experts on the lower convex hull of the bin's cost-error plane (which is why Figure 3 looks the way it does). However, the paper's central motivation is OOD generalization. in the Navier-Stokes experiments, the training set uses viscosities in {3.2, 1.6, 0.8} x 10^-2, while the test set evaluates viscosities in {0.8, 0.4, 0.2, 0.1} x 10^-2 (and similarly disjoint buoyancy values {0.05, 0.55} in Section 4.3). Because the error table E is populated during training, strictly OOD test parameters fall into bins that were never seen during training. If those bins use training statistics, the pure neural operator's error is severely underestimated (for example, UNO-128 has a training MSE of 0.26 x 10^-2 vs. a test MSE of 17.0 x 10^-2), which would cause the LP to route OOD samples to the exact neural operator that fails out of distribution. Conversely, if the table is populated using validation or test errors, that constitutes direct test-set leakage. Either way, there is no valid mechanism to route unseen OOD regimes.

  2. Essential baselines are omitted Within each bin, MF-MoE's expected MSE is simply a convex combination of its constituent experts. Yet Table 2 and Table 3 only report raw upsampled solver outputs, completely omitting the test MSE of the individual hybrid experts (Hybrid-Coarse, Hybrid-Medium, Hybrid-Fine). Without seeing the standalone hybrid accuracies, it is impossible to evaluate the router. The most critical error is perhaps every hybrid expert included in the MoE pool costs between 12s and 38s, violating the c = 10.0s budget on every single call. The budget is only met in expectation by mixing with the 0.2s neural operator, which contradicts the paper's repeated claims of strictly adhering to the computational budget. However, Table 1 shows that the XCoarse (16 x 16) solver takes 8.0s, which is under the 10s limit. The authors compare against the raw XCoarse solver (15.1 x 10^-2), but exclude a Hybrid-XCoarse (XCoarse solver + Neural Operator) baseline. A standalone Hybrid-XCoarse model deterministically satisfies the 10s budget on every sample without needing MoE routing.

  3. Flawed hybrid architecture and mathematical errors in the method description As shown in Figure 7 (Appendix A.1), the neural operator branch inside the hybrid expert only receives the initial conditions and physics vector; it never takes the coarse solver's prediction as input. A neural network cannot learn to correct localized numerical or interpolation errors if it is completely blind to the coarse solution. It is merely an independent pure neural operator whose output is blended with the upsampled solver output via y^out=αy^f+(1−α)Δy\hat{y}_{out} = \alpha \hat{y}_f + (1-\alpha)\Delta y, where the hyperparameter α\alpha is never even reported. The mathematical description in Section 3.3 is also flawed. In Equation 5, the running average update at iteration t for bin k is written as Ek,i(t)=1kEk,i(t−1)+k−1k∥Eθi(x)−y∥2E_{k,i}^{(t)} = \frac{1}{k} E_{k,i}^{(t-1)} + \frac{k-1}{k} \|E_{\theta_i}(x) - y\|^2. This confuses the bin index k with the sample count, and inverts the averaging weights, giving weight (k-1)/k to the new observation and zero weight to new observations in bin k=1. Similarly, Equation 4 sums over NtrainN_{train} but normalizes by ∣N∣|N|. In addition, the timing model also appears dominated by overhead: per frame, the Fine solver (128 x 128, 8 steps) performs 512 times more cell updates than the XCoarse solver (16 x 16, 1 step), yet takes only 4.75 times longer (38s vs. 8s). Taking 8 seconds for a single explicit Euler step on a 16 x 16 grid indicates the timings measure Python or I/O overhead rather than actual numerical solve time.

  4. Toy dataset scale and severe numerical contradictions across tables and figures Appendix A.2 reveals that the Navier-Stokes training set contains only 60 input-output pairs (5 trajectories x 3 viscosities, length 8), and the test set has 80 pairs. Models with 67M to 440M parameters are trained for 8,000 samples by repeatedly reusing these 60 pairs, with no random seeds or error bars reported. Moreover, the viscosity 0.8 x 10^-2 appears in the train, validation, and test splits simultaneously, and the standalone FNO baselines perform worse than or close to predicting a constant mean field (Gaps to Means is roughly 0.19 in Figure 2, while FNO64-4 gets 0.203).

Questions for authors
  1. How are test samples with OOD parameter values (viscosities in {0.4, 0.2, 0.1} x 10^-2 and buoyancies in {0.05, 0.55}) mapped to error statistics in the look-up table, and on what data split are those statistics computed?
  2. What are the test MSEs of each standalone hybrid expert (Hybrid-Coarse, Hybrid-Medium, Hybrid-Fine) and of a budget-feasible Hybrid-XCoarse (8s) model?
  3. Why does the neural operator branch in the hybrid model (Figure 7) not take the coarse solver prediction as input? What value of α\alpha is used to combine them?
  4. Why do the neural operator MSEs in Figures 4 and 5 (0.3 to 1.1) differ by an order of magnitude from the corresponding values in Tables 2 and 3 (0.08 to 0.20), and why is the 4-step rollout error in Figure 6 lower than the 1-step error in Table 2?
  5. Why does a single step of explicit Euler on a 16 x 16 grid (XCoarse) take 8.0 seconds, and is the coarse solver re-initialized from the current fine input state at every prediction step?
Reviewer_irSJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Multi-Fidelity Mixture-of-Experts (MF-MoE) framework that combines a fast neural operator with solver-based hybrid models at different resolutions. A routing strategy uses historical error and runtime statistics, discretized by physical parameters, to select experts under a user-specified time budget. Experiments on Navier-Stokes flows, buoyancy variations, multi-step prediction, and a Burgers-equation dataset suggest that the framework can improve prediction accuracy over standalone neural operators while controlling average inference cost. The problem is practically relevant, and the paper makes a useful attempt to connect model accuracy with runtime constraints.

Critical strengths
  1. The paper addresses a real tension between fast neural surrogates and more robust, but expensive, solver-based models. Making runtime budget an explicit part of routing is a useful direction.
  2. The authors test multiple operator backbones and solver fidelities, and extend the study beyond the primary Navier-Stokes setting. The paper also provides substantial implementation and dataset details in the appendices.
Critical weaknesses
  1. The runtime guarantee is unclear. The runtime target appears to guide operator selection rather than enforce a strict per-input budget. The paper should clarify what happens when the selected operator exceeds the target and whether the reported constraint is guaranteed in deployment.

  2. The gating strategy may not generalize reliably. Operator selection is based on discretized physical-parameter bins, but the paper does not sufficiently explain how this works for inputs near bin boundaries or for parameter values outside the sampled range.

  3. The evaluation does not yet establish robust end-to-end benefits. The runtime accounting may omit inference overhead, while the experimental discussion leaves important aspects of the data protocol and hybrid correction unclear. This makes it difficult to judge whether the reported gains would hold in practical deployment.

  4. Equation (5) is ambiguous. The stated running-average update is difficult to reconcile with the equation; the roles of k and t should be defined more precisely.

  5. Discretization choices need justification. The paper should explain how the number and boundaries of parameter bins were selected, and whether the results are sensitive to these choices.

  6. The data split and sample reuse should be reported more explicitly. Clarifying overlap between splits and the amount of unique training and evaluation data would improve reproducibility.

  7. The ablation study is limited. Additional ablations would help isolate the contributions of the gating strategy, operator selection, and hybrid correction.

Questions for authors

The paper presents a relevant idea and a promising framework for managing accuracy-runtime trade-offs. However, the central claims depend on an expected-cost formulation, coarse parameter-conditioned statistics, and runtime estimates whose deployment scope is unclear. The limited data, ambiguous update rule, and lack of uncertainty reporting also make it difficult to judge how robust the results are.

Reviewer_hHoWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Multi-Fidelity Mixture of-Experts (MF-MoE), a framework which tackles the dilemma between accuracy gains and computational costs of hybrid neural PDE solvers. It does so by learning a set of experts composed of a pure neural PDE solver (the cheapest and quickest) and hybrid PDE solvers each with decreasing granularity (increasingly costly and accurate), as well as a router that generates a probability distribution over these experts for sampling one on a given trajectory. The router uses the physics parameters of the trajectory at hand along with the statistics (average error and computation time) of each expert aggregated for these specific parameters.

Critical strengths
  • The paper is easy to follow;

  • The experiments quantifying the impact of time budget and physics parameters on the prediction error, justifying the relevance of different expert models for different regimes or time constraints.

Critical weaknesses
  • As stated in Eq. 3 and note 2 of Table 2, only the average of the computation time is used to determine if the time constraint is violated or not, which does not prevent it from being violated on a sample-wise basis ; more generally, at a sample-wise level, the method does not solve the dilemma between an accurate but costly fine grid hybrid model and a less accurate but quicker pure neural PDE solver, as the router still has to choose a single model to perform the prediction. The advantage of using one or the other randomly only appears on average, and is only expected to happen ; if you have model AA : costly and accurate and model BB : light but inaccurate, then randomly sampling AA or BB will be both less costly than AA and more accurate than BB on average, but half of the predictions are slow, and the other half are inaccurate. I thus suspect the variance of the error and time cost to be higher for this method than for pure alternatives, which I don't think is a desirable property.

  • Overall the main contribution is the router, not the expert models it picks from, and, in my opinion, it simply balances time and error constraints lazily (because balanced on average) to the cost of a higher discrepancy in result's quality, and the impact of physics parameters (which could be interesting in a context of multiple experts for different bins) is too low / not thoroughly assessed.

Questions for authors
  • I see in Table 2 pure numerical solvers, pure neural solvers and your method ; your method uses hybrid neural solver, which use a numerical solver conjointly with a neural network : do you have results for these hybrid models ?
  • in Table 2, the "< 10 sec ?" column hides how the time cost scales with each method, could you add an additional column with the actual average time cost value ?
  • could you evaluate the variance of the MSE and time cost for Table 2 ?
  • in Figure 3 could you clarify what each expert corresponds to (neural, coarse, fine) ? it is also unclear what each bin corresponds to (which range of viscosity for each)
  • in Figure 3 still do you know why expert 4 is absent of bin 1 ? does it have a high error for this parameterization of the equation ?
  • as I understood the router uses a history profile of EE and CC for each parameter bin to solve a linear programming problem, but it is unclear how it is supposed to have an history on unseen parameters bin at validation and test as {0.4,0.2,0.1}×10−2\{0.4, 0.2, 0.1\} \times 10^{-2} is unseen during training

Comments

  • Figure 4 : shouldn't the title be "Impact of the Time Budget c on the MSE" rather than "on the Gating Mechanism" ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.