acceptodds
Under review as a conference paper at ICLR 2027

HotPOT: Operator Transformer with Heterogeneous Physics Experts for Large-Scale PDE Pre-training

Abstract

Large-scale PDE pre-training trains a single model on the data from multiple partial differential equations (PDEs). However, adding heterogeneous PDEs with different governing equations, physical variables, and variable interactions does not always improve predictive performance across tasks. Existing PDE mixture-of-experts models mitigate cross-task interference through learned routing, but typically use homogeneous expert architectures, making it difficult to directly reflect differences in state evolution across equations. We introduce HotPOT, an operator transformer with heterogeneous physics experts for large-scale PDE pre-training. HotPOT uses a shared encoder to capture information common across PDEs, while each PDE family uses a neural–numerical expert that pairs a learnable predictor with a fixed numerical update containing no trainable parameters. Each predictor outputs the quantities required by its paired numerical update, which then computes the next state. This design allows different PDEs to share a common encoder while retaining their own learnable predictors and numerical updates. Experiments across five PDE families and six settings show that HotPOT outperforms MoE-POT in all 24 matched-scale comparisons before and after fine-tuning, reducing rollout error by 50.7% on average. Notably, the 17M model also surpasses substantially larger MoE-POT models, and the performance gains extend to zero-shot prediction under unseen physical conditions. Code is available at https://anonymous.4open.science/r/HotPOT-code-1727/.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Partial Differential Equations, PDE Foundation Models, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7FZ8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes HotPOT, a multi-PDE pre-training framework combining a shared history encoder with PDE-specific neural–numerical experts. Each expert pairs a learned predictor with a fixed numerical update to handle variable coupling and state evolution for different equations. The model is jointly trained using next-state supervision and predicts subsequent physical fields autoregressively. The paper evaluates HotPOT against MoE-POT and other baseline methods using relative L2 error across six settings spanning five PDE families, with promising results.

Critical strengths
  1. The separation between shared field-history representations and PDE-specific predictor–update pairs is a reasonable approach to heterogeneous PDE learning.
  2. The SWE and CNS update constructions also provide explicit conservation and admissibility properties under the stated assumptions.
  3. The evaluation includes multiple PDE families, architectural controls, and transfer to an unseen family.
Critical weaknesses
  1. Insufficient evidence of reproducibility and lack of compute-matched comparisons. Although the paper provides training and evaluation settings, it does not establish sufficiently verifiable correspondence between the processed data, split manifests, model checkpoints, and reported results. Training code alone is insufficient for independent verification. Moreover, matching activated-parameter counts does not imply comparable computational costs: HotPOT involves repeated readouts and numerical operations, and stores substantially more parameters than the corresponding MoE-POT models. Appendix A.5 acknowledges the absence of runtime comparisons under identical conditions, making it difficult to separate architectural benefits from the effects of additional capacity and computation.
  2. Insufficient validation of baseline comparability and result robustness. Table 1 imports baseline results from MoE-POT without demonstrating that all methods were evaluated through a unified pipeline. For example, the fine-tuned SWE errors for DPOT-T/S/M are 0.00313/0.00315/0.00260, compared with 0.00280/0.00224/0.00218 in the original paper. Different pre-training settings may explain these discrepancies, but protocol clarification and controlled comparisons are still needed. Furthermore, task-specific learning-rate selection and sampling adjustments make it difficult to distinguish the contributions of model design from those of training strategies.
Questions for authors
  1. What exact artifacts allow an independent reviewer to reproduce Table 1 without repeating full pre-training? Please identify the processed-data sources or deterministic preparation scripts, exact split manifests, evaluated checkpoints, and evaluation commands.
  2. Can the authors reconcile the imported DPOT results with the original DPOT Table 1, specifying differences in pre-training data, preprocessing, checkpoint selection, rollout horizons, and metric aggregation? Have the imported baseline scores been verified using the same evaluation pipeline as HotPOT?
  3. How do HotPOT and MoE-POT compare under matched computational budgets? Please report training GPU-hours, inference latency or throughput, peak memory, and the actual NS substep/readout counts on identical hardware.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 7FZ8 (1/4)

We sincerely thank the Reviewer for the precise comments and questions. The revised parts are highlighted in blue in the uploaded revised manuscript.

Brief Response:

  • W1 / Q1. The code, the checkpoints, the test indices and the evaluation commands are released, so Table 1 can be reproduced without any training. The Direct-state ablation over three seeds and the measured cost separate the gain of the design from extra capacity and computation.
  • W2 / Q2. The baseline rows come from MoE-POT Table 1. We re-evaluated the released DPOT-T and MoE-POT-T checkpoints through our pipeline, with a deviation below 0.8% from the cited values.
  • Q3. At a matched computational budget, HotPOT-T needs 24% fewer GPU-hours for pre-training, and its inference time and memory are on par with MoE-POT-T.

W1a / Q1. Reproducibility

W1. Insufficient evidence of reproducibility and lack of compute-matched comparisons. Although the paper provides training and evaluation settings, it does not establish sufficiently verifiable correspondence between the processed data, split manifests, model checkpoints, and reported results. Training code alone is insufficient for independent verification.

Q1. What exact artifacts allow an independent reviewer to reproduce Table 1 without repeating full pre-training? Please identify the processed-data sources or deterministic preparation scripts, exact split manifests, evaluated checkpoints, and evaluation commands.

Everything the reviewer lists is now released, and Table 1 can be reproduced without any training:

  • Checkpoints. https://anonymous-hf.com/a/ryou1c8aizn3/ holds the fourteen checkpoints behind the 24 HotPOT entries of Table 1. They are HotPOT-T and HotPOT-S after pre-training and the twelve fine-tuned models.
  • Correspondence with Table 1. docs/CHECKPOINTS.md and metadata/checkpoints.json list, for each of the 24 entries, the checkpoint file with its SHA-256, the test-row indices, the number of history frames and rollout steps, the expected error to six digits and the exact evaluation command.
  • Data and splits. The processed data follow the MoE-POT repository, as documented in docs/DATA.md, and the data preparation script is included. The CFDBench test rows and the wave split are stored as manifests in metadata/.
  • Code and command. https://anonymous.4open.science/r/HotPOT-BA73/ contains the training, fine-tuning, evaluation and data preparation code. One entry of Table 1 is reproduced by python evaluate.py --workspace ./workspace --checkpoint ./workspace/checkpoints/HotPOT-T_pretrain_E1000.pt --scale 17 --task ns2d_fno_1e-5 --batch-size 4 --num-workers 0.

We verified this correspondence ourselves. We re-evaluated all 24 entries on a different GPU (6,880 trajectory scores), and every entry agrees with the paper at the printed precision.

Revision location: The released materials are listed in the reproducibility statement and in Appendix A.17.

AuthorsOpenReview ↗

Response to Reviewer 7FZ8 (2/4)

W1b. Computation, capacity and storage

Moreover, matching activated-parameter counts does not imply comparable computational costs: HotPOT involves repeated readouts and numerical operations, and stores substantially more parameters than the corresponding MoE-POT models. Appendix A.5 acknowledges the absence of runtime comparisons under identical conditions, making it difficult to separate architectural benefits from the effects of additional capacity and computation.

Both have now been separated:

  • Computation. We measured it on the same hardware. Over the complete rollouts of the six settings, HotPOT-T takes 1% less time than MoE-POT-T (3297 ms against 3331 ms), and its peak memory is on par (184 to 185 MiB against 182 MiB). The details are in Q3.
  • Capacity. The paper contains the control for this question. The Direct-state ablation of Table 3 has family-private predictors, exactly the same stored parameters as HotPOT-T (97,902,720) and the same training protocol. The only change is that the numerical update is replaced by direct prediction of the next state. To make the comparison robust, we trained both models with two further seeds. The table gives the mean and standard deviation over three seeds on the settings whose expert uses a numerical update.
SettingDirect state (same capacity, no numerical update)HotPOT-TReduction
NS ν=1e-50.0527 ± 0.00020.0316 ± 0.000240%
NS ν=1e-30.00657 ± 0.000630.00318 ± 0.0000952%
CNS0.0106 ± 0.00070.00715 ± 0.0003832%
SWE0.00188 ± 0.000040.00179 ± 0.000025%
DR0.120 ± 0.0160.0211 ± 0.006182%

On each of these settings, every HotPOT-T seed is below every Direct-state seed. With the capacity held equal, the gain comes from the numerical updates. CFDB is not listed because its update is a plain assignment, so there is no numerical update to test.

  • Storage. The stored totals count all experts together, but a task uses only the shared encoder and the expert of its own family. To deploy one NS, CNS, SWE or DR task, HotPOT stores 16.8 to 17.2M parameters at the T scale, against 30.2M for MoE-POT-T. MoE-POT cannot be reduced in this way, because its routing depends on the input. The single-expert files are released in the folder single_expert/ of https://anonymous-hf.com/a/ryou1c8aizn3/, and their predictions are bit-identical to those of the full models.

Revision location: Section 5.2 reports these reductions, and Appendix A.14 gives the results of every seed. Section 5.1 and Appendix A.11 report the single-task storage.

AuthorsOpenReview ↗

Response to Reviewer 7FZ8 (3/4)

W2a / Q2. Comparability of the baselines and robustness

W2. Insufficient validation of baseline comparability and result robustness. Table 1 imports baseline results from MoE-POT without demonstrating that all methods were evaluated through a unified pipeline. For example, the fine-tuned SWE errors for DPOT-T/S/M are 0.00313/0.00315/0.00260, compared with 0.00280/0.00224/0.00218 in the original paper. Different pre-training settings may explain these discrepancies, but protocol clarification and controlled comparisons are still needed.

Q2. Can the authors reconcile the imported DPOT results with the original DPOT Table 1, specifying differences in pre-training data, preprocessing, checkpoint selection, rollout horizons, and metric aggregation? Have the imported baseline scores been verified using the same evaluation pipeline as HotPOT?

The DPOT rows of our Table 1 are not taken from the original DPOT paper [1]. They are the DPOT models that the MoE-POT authors pre-trained again on the six-dataset mixture and reported in MoE-POT Table 1 [2], and we take all baseline rows from that table. The original DPOT was pre-trained on twelve datasets. This is the reason for 0.00313 / 0.00315 / 0.00260 against 0.00280 / 0.00224 / 0.00218 on fine-tuned SWE. The table lists the items the reviewer asks about.

Original DPOT [1]MoE-POT benchmark [2] (source of our baseline rows)HotPOT
Pre-training data12 datasets6 datasetsthe same 6 datasets
Processed dataits ownMoE-POT repositoryMoE-POT repository
Pre-training1000 epochs1000 epochs1000 epochs, final checkpoint
Fine-tuning200 or 500 epochs200 epochs200 epochs, final checkpoint
Test setsits own200 / 200 / 200 / 60 / 60 / 1000 trajectoriesidentical
Rollout horizon and metricfull-horizon relative L2full-horizon relative L2the same released metric

The conclusion is the same under both sets of numbers. On fine-tuned SWE, HotPOT-T reaches 0.00162, which is below all six DPOT values quoted by the reviewer.

Yes, the imported scores have been verified with the same evaluation pipeline as HotPOT. We evaluated the checkpoints released by the MoE-POT authors (DPOT-T and MoE-POT-T, pre-trained) with exactly this pipeline, on all six settings (12 entries, 3,440 trajectory scores). The largest deviation from the cited value is 0.353% for DPOT-T and 0.792% for MoE-POT-T, so our pipeline reproduces the cited numbers. The other baseline entries have no public checkpoints, and the paper marks them as taken from MoE-POT Table 1.

With these per-trajectory scores we also tested the robustness of the comparison. The table gives the error reduction of HotPOT-T after pre-training with 95% paired bootstrap intervals over test trajectories (20,000 resamples), and the number of test trajectories on which HotPOT-T is better.

Setting (test trajectories)vs MoE-POT-Tbetter onvs DPOT-Tbetter on
NS ν=1e-5 (200)53.9% [52.8, 54.9]200/20070.6% [70.0, 71.3]200/200
NS ν=1e-3 (200)57.7% [55.5, 59.7]200/20079.0% [77.9, 80.1]200/200
CNS (200)35.9% [32.9, 38.7]185/20059.9% [57.8, 62.0]200/200
SWE (60)72.6% [70.6, 74.5]60/6071.9% [69.8, 74.0]60/60
DR (60)63.6% [62.0, 65.1]60/6074.0% [72.8, 75.1]60/60
CFDB (1000)58.7% [51.6, 64.4]989/100067.5% [61.9, 72.1]993/1000

We also trained HotPOT-T with two further seeds. Every seed is below MoE-POT-T on every setting, by at least 29%.

Revision location: Section 5 and the caption of Table 1 state the source of every baseline row. Appendix A.12 adds the original DPOT numbers, the re-evaluation and the bootstrap details, and Appendix A.14 gives the results of every seed.

Reviewer_nSbbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HotPOT, a model created to address cross-task interference caused by different control equations, physical variables, and variable coupling in various physical systems. HotPOT uses a historical encoder to find shared spatiotemporal features across different PDEs and assigns specific physical experts to each PDE. As a result, the model can predict the right physical quantities and update states as each system evolves. Experiments show that HotPOT performs better than the current PDE foundation model.

Critical strengths
  1. Unlike traditional PDE hybrid expert models that use similar experts and routing, HotPOT combines predictions from different physical systems using a numerical update method. It keeps a shared spatio-temporal representation while maintaining the unique evolution of each PDE. This reduces interference between different physical tasks and uses fixed numerical programs to capture the specific constraints of each PDE. This modeling approach includes physical priors for joint pre-training of multiple PDEs.
  2. The experiments are thorough and show that the method is effective and somewhat transferable. The paper compares pre-training and fine-tuning results for different model sizes, examines the impact of shared encoders and neural-numerical update methods through ablation studies, and looks at zero-shot prediction, few-shot adaptation, and transfer to new PDE types in unfamiliar conditions. Importantly, the smaller HotPOT model performs better than the larger MoE-POT in several cases, suggesting that better results do not just come from having more parameters and offering useful insights.
Critical weaknesses
  1. The method depends a lot on defining physical systems in advance and does not yet solve the automatic combination of unknown multi-physics fields. HotPOT needs users to set up specific predictors and fixed numerical update steps for each PDE family, and adding new PDE families means creating new update steps. For instance, in combustion problems, there are several physical mechanisms working together, but current methods still cannot automatically identify each mechanism or build the right coupling relationships. For complex systems with unknown equations or coupling methods not seen during training, it is still unclear how well the method can generalize.
  2. As more PDE families are added to the model, using system-specific predictors will require more storage, and multi-stage numerical updates could make inference more expensive. Also, the main experiments in the paper only cover five types of PDEs on regular grids. While the author do test migration to new PDE families, these experiments are not enough to show that the method can be easily applied to more complex geometries, unknown coupled structures, or multiphysics systems.
Questions for authors
  1. The paper creates dedicated predictors and fixed numerical update procedures for different PDE families. However, real physical systems often involve several coupled mechanisms at once, like combustion and chemical reactions. Can HotPOT handle a case with multiple coupled PDEs directly? If not, does it require combining several experts, and how to ensure that variable exchange and physical constraints remain consistent across them?
  2. When applying this method to a new family of PDEs, it seems necessary to train a new predictor and set up the fixed-value update procedure. This suggests the model still depends on some prior physical knowledge. Could you clarify which equation details or numerical update rules must be provided in advance? Also, if the PDE type is unknown during testing, or if a case involves several physical mechanisms not included in training, can the model automatically identify and combine the right experts? If so, could you share more experiments that show this?
  3. The main experiments in the paper focus on five types of PDEs and regular grids. The author notes that adding dedicated experts increases storage needs, and multi-stage updates may raise inference costs. Could you explain how computational and storage costs change as the model is expanded to more PDE families or complex geometries? Has you looked at how these factors affect the model’s practical use and its ability to generalize to a wider range of physical systems?
9 comments
AuthorsOpenReview ↗

Response to Reviewer nSbb (1/4)

We sincerely thank the Reviewer for the constructive comments and questions. The revised parts are highlighted in blue in the uploaded revised manuscript.

Brief Response:

  • W1. A new PDE family does not need a new update. When nothing is known about a system, the generic update is used unchanged.
  • Q1. One expert handles a complete coupled system. On simulated combustion with 15 coupled fields, the error of HotPOT is 17 to 42% lower than that of MoE-POT-T on held-out operating conditions.
  • W2 / Q3. The storage and the compute of a task do not grow with the number of families, and we measured storage, inference time and memory.
  • Q2. No equation detail must be provided for a new family. Known structural properties, such as conservation laws, are built into the update when they are available, and a system of unknown type uses the generic update.

W1a. Predefined systems and new update steps

W1. The method depends a lot on defining physical systems in advance and does not yet solve the automatic combination of unknown multi-physics fields. HotPOT needs users to set up specific predictors and fixed numerical update steps for each PDE family, and adding new PDE families means creating new update steps.

A new family does not need a new update step. The design is flexible about how much physics is supplied:

  • When nothing is known about a system, the generic update is used unchanged. The predictor outputs the increment of the state and the update adds it, so it contains no information about the equation. In the paper, the wave equation is a family that is absent from pre-training, and it is added in this way (Section 5.4). It required only the list of observed variables and one new predictor on top of the pre-trained encoder.
  • When structural properties of the solution are known, such as a conserved total or a positivity bound, they can be built into the update, as for CNS and SWE.

What is pre-trained and shared is the history encoder. This is the common paradigm of foundation models, in which a pre-trained backbone is combined with task-specific modules [3, 4].

W1b. Unknown equations and couplings not seen during training

For instance, in combustion problems, there are several physical mechanisms working together, but current methods still cannot automatically identify each mechanism or build the right coupling relationships. For complex systems with unknown equations or coupling methods not seen during training, it is still unclear how well the method can generalize.

Combustion is such a system, and HotPOT generalises to it:

  • Its coupling between flow, heat and chemistry does not appear in pre-training, and no information about its equations is given to the model.
  • The mechanisms do not have to be identified one by one, because the predictor of the new family learns the complete coupling from data.
  • With the pre-trained encoder, the error of HotPOT is 17 to 42% lower than that of MoE-POT-T on operating conditions that were not seen in training (Q1, Appendix A.15).
AuthorsOpenReview ↗

Response to Reviewer nSbb (2/4)

Q1. Coupled multi-physics

Q1. The paper creates dedicated predictors and fixed numerical update procedures for different PDE families. However, real physical systems often involve several coupled mechanisms at once, like combustion and chemical reactions. Can HotPOT handle a case with multiple coupled PDEs directly? If not, does it require combining several experts, and how to ensure that variable exchange and physical constraints remain consistent across them?

Yes, it handles them directly, and no combination of experts is needed. One expert corresponds to one complete coupled system. It works in three steps:

  • Encode. The shared history encoder reads every field with the same weights, so it applies to any number of variables.
  • Couple. A coupling module inside the expert mixes the features of all variables. The variable exchange happens here, inside one network, so there are no separate solvers to synchronise.
  • Update. One update advances all variables together, and physical constraints are imposed on the joint state. In the CNS expert, the conservation of mass, momentum and energy and the positivity of density and pressure are enforced on all four variables at once.

To answer this question on the system the reviewer names, we added a combustion experiment. It uses the simulated combustion data of RealPDEBench [5]: NH3/CH4 flames with 15 coupled fields (pressure, heat release, seven species, temperature and velocity), in which flow, heat and chemistry are coupled in one state. The combustion expert follows the same three steps. It reuses the pre-trained shared encoder, adds one new coupling module and predictor, and applies the generic update. No information about the combustion equations or the operating condition is used, so the coupling between flow, heat and chemistry is learned from data.

MoE-POT-T is fine-tuned on the same data with the same protocol, and both models are evaluated on operating conditions that are held out from training. The table gives the rollout error over 1 to 20 frames.

1 frame5 frames10 frames20 frames
MoE-POT-T0.05450.1480.2720.430
HotPOT0.03160.1160.2230.358
Reduction with 95% interval42% [40, 45]22% [17, 26]18% [14, 22]17% [14, 20]

HotPOT is more accurate than MoE-POT-T on every held-out condition at every horizon.

Revision location: Appendix A.15 gives the detailed results and experimental details, and the code is in experiments/combustion/ of the repository.

AuthorsOpenReview ↗

Response to Reviewer nSbb (3/4)

W2a / Q3. Storage and compute as families are added

W2. As more PDE families are added to the model, using system-specific predictors will require more storage, and multi-stage numerical updates could make inference more expensive.

Q3. The main experiments in the paper focus on five types of PDEs and regular grids. The author notes that adding dedicated experts increases storage needs, and multi-stage updates may raise inference costs. Could you explain how computational and storage costs change as the model is expanded to more PDE families or complex geometries? Has you looked at how these factors affect the model’s practical use and its ability to generalize to a wider range of physical systems?

The storage needed to use the model on a task does not grow with the number of families. A task uses only the shared encoder and the expert of its own family. The other experts can be deleted without changing a single output. Adding a family adds one expert file to the library and leaves the existing tasks unchanged. The table gives the number of parameters that must be stored to deploy one NS, CNS, SWE or DR task.

Model scaleHotPOTMoE-POT
T16.8 to 17.2M30.2M
S89.6 to 91.2M166M
  • The same holds for a new family. The combustion model of Q1 stores 18.6M parameters, against 30.2M for MoE-POT-T on the same task.
  • MoE-POT [2] cannot be reduced in this way. Its routing depends on the input, so all of its experts must be kept even for a single task.
  • To make this checkable, we release the single-expert files of the fine-tuned models at https://anonymous-hf.com/a/ryou1c8aizn3/ in the folder single_expert/. Their predictions are bit-identical to those of the full models.

The compute cost does not grow with the number of families either. A prediction executes the shared encoder and one expert, whatever the number of families in the model. The stages of the numerical update are also cheap: the history encoder, the blocks of the expert and the decoder run once per predicted step, and only the light readout is repeated in each stage. To address this concern, we measured the cost on one H100 in FP32.

HotPOT-TMoE-POT-T
Total time for the complete rollouts of the six benchmark settings3297 ms3331 ms
Peak memory (NS, CNS, SWE and DR tasks)184 to 185 MiB182 MiB

HotPOT-T takes 1% less time than MoE-POT-T, and its memory is on par. The time covers the complete model execution, including all stages of the numerical updates, and the memory is measured with the shared encoder and the expert of the task loaded.

Revision location: Section 5.1 reports the single-task storage, and Appendix A.11 gives the storage and the measurement details.

W2b. Coverage of the experiments

Also, the main experiments in the paper only cover five types of PDEs on regular grids. While the author do test migration to new PDE families, these experiments are not enough to show that the method can be easily applied to more complex geometries, unknown coupled structures, or multiphysics systems.

The five families and six settings are exactly the standard benchmark setting of DPOT [1] and MoE-POT [2], and we follow it so that the comparison is like for like. Beyond this benchmark, the paper and the revision test the cases the reviewer names:

  • Complex geometries. CFDBench [6] contains three geometries (cavity, cylinder and tube), and the geometry is given to the model as a mask. On six CFDBench groups with unseen boundary conditions, geometries and fluid properties, the zero-shot error of HotPOT-T is 2.5 to 6.7 times lower than that of MoE-POT-T in every group (Section 5.3, Table 4). The cost on a new geometry depends on the grid size and the number of variables, and not on the number of families in the model.
  • New PDE families. The wave equation is absent from pre-training and is added with the generic update. With 16 / 32 / 64 / 128 training trajectories, HotPOT reaches 0.506 / 0.395 / 0.304 / 0.220 against 0.576 / 0.517 / 0.435 / 0.365 for fully fine-tuned MoE-POT-T (Section 5.4, Table 11).
  • Unknown couplings and multi-physics. The combustion experiment of Q1 couples flow, heat and chemistry in 15 fields, and this coupling does not appear in pre-training. On operating conditions that were not seen in training, the error of HotPOT is 17 to 42% lower than that of MoE-POT-T.

In practical use (Q3), a new system therefore costs one new expert, and the storage, time and memory of the existing tasks stay the same. The interface between predictor and update does not depend on the grid, so native irregular meshes and 3D change the backbone and keep the design. Section 6 of the revision lists them as future work.

Reviewer_nYtCOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes HotPOT, an operator transformer for multi-PDE pre-training. A shared field-wise history encoder (patch encoding, temporal aggregation and AFNO-style blocks, inherited from DPOT/MoE-POT) feeds one private expert per PDE family. Each expert has a learned predictor and a hand-written numerical update with no trainable parameters: Strang/midpoint integration for NS, a limiter-based conservative update for CNS, a mean-centered height increment for SWE, and so on. Selection is by dataset identity, with no router. On six settings from the MoE-POT benchmark, HotPOT reduces rollout error by 50.7% on average against MoE-POT at matched activated parameters. It also reports gains in zero-shot, few-shot and wave-equation transfer.

Critical strengths
  1. Embedding known, PDE-specific update structure in the model is sensible.
  2. The reported gains over MoE-POT are large and consistent on the chosen benchmark.
Critical weaknesses
  1. Limited novelty. The encoder, time aggregation, and Fourier blocks come from DPOT and MoE-POT. Using a learned predictor with a fixed numerical update is an old idea (solver-in-the-loop, FINN, OpsSplit, Origo). A shared encoder with one head per task is standard multi-task learning. The “experts” are chosen by dataset name, with no router, so the MoE framing is too strong. The conservation proofs are correct but very simple.
  2. The design goes against the goal of pre-training. Each new PDE needs a hand-written update and a new predictor. A pre-trained model should help with new equations, but here the manual work grows with every family. The only new-family test is the wave equation, and it also needs a manual setup.
  3. Small experiments. There are only six datasets from five PDE families, all on 2D regular grids. The test sets are small (60–200 trajectories). All results use one seed. Some ablation gaps are tiny (0.0101 vs 0.0104) and have no error bars. HotPOT-S is worse than HotPOT-T on DR, and this is not explained.
  4. The cost comparison is misleading. “17M beats 288M” counts only activated parameters. The stored size is 98M vs 30M (T scale) and 518M vs 166M (S scale). No runtime or memory is reported.
  5. The source of the gain is unclear. HotPOT differs from MoE-POT in several ways at once: larger private predictors, no router, and the numerical update. The only clean test of the update is “Generic iteration”, and it covers NS only (gain 1.5% and 19.7%). A baseline with per-family heads of the same stored size, but no numerical update, is missing. So the gain may come from extra capacity, not from the physics update.
  6. Missing baselines and unequal protocols. Walrus, Poseidon, Unisolver, and others are not compared. In few-shot tests, HotPOT tunes its learning rate, but MoE-POT uses a fixed one.
Questions for authors
  1. What are the inference time and memory against MoE-POT at the same activated and the same stored size?
  2. Does a MoE-POT/DPOT with private per-family heads of equal stored size, but no numerical update, match HotPOT?
  3. Can the update be chosen or learned for an unseen PDE? What happens if the wrong update is used?
  4. Can you report multiple seeds and comparisons with Walrus/Poseidon/Unisolver, and explain why the S model underperforms the T model on DR?
  5. How do you extend this method on irregular meshes, 3D or higher-Reynolds settings?
11 comments
AuthorsOpenReview ↗

Response to Reviewer nYtC (1/8)

We sincerely thank the Reviewer for the detailed comments and questions. The revised parts are highlighted in blue in the uploaded revised manuscript.

Brief Response:

  • W1. Earlier neural-numerical methods use one template for one system or for every system. In HotPOT each PDE family has its own predicted quantity and its own update inside one pre-trained model.
  • W2. The pre-trained encoder transfers to unseen conditions and to two families absent from pre-training, with a generic update that needs no term of the equation.
  • W3. Paired bootstrap intervals and three training seeds now support the comparison, and every seed is at least 29% below MoE-POT-T on every setting.
  • W4 / Q1. We measured inference time, memory and storage at the same activated size and at the same stored size.
  • W5 / Q2. The baseline described as missing is the Direct-state ablation of Table 3, and three seeds confirm that the gain comes from the numerical updates.
  • W6. We added the Poseidon architecture as a baseline and selected the few-shot learning rates of both models from the same candidates.
  • Q3. An unseen PDE uses the generic update. A new experiment shows that an update with a false physical constraint destroys the accuracy.
  • Q4. We report three seeds and the comparison with Poseidon, and we explain the result of HotPOT-S on DR.
  • Q5. We discuss the extension to irregular meshes, 3D and higher Reynolds numbers.
AuthorsOpenReview ↗

Response to Reviewer nYtC (2/8)

W1. Novelty

W1. Limited novelty. The encoder, time aggregation, and Fourier blocks come from DPOT and MoE-POT. Using a learned predictor with a fixed numerical update is an old idea (solver-in-the-loop, FINN, OpsSplit, Origo).

We respectfully disagree that HotPOT is the "old idea" of a learned predictor with a fixed numerical update:

  • Earlier neural-numerical methods build one template for one physical system [3, 4, 5].
  • Multi-PDE models such as Origo [6] and DISCO [7] pre-train across PDEs but apply one template to every system.
  • In HotPOT each PDE family has its own predicted quantity and its own parameter-free update inside one pre-trained model. All of them are trained jointly through one shared history encoder, from next-state supervision only.
MethodWhat one model coversWhat the network outputsNumerical partOne template for every PDE
Solver-in-the-loop [3]one system with its existing solvera correction to the solver stepa complete classical solversingle system
FINN [4]one equation per modelflux and reaction termsfinite-volume stencil and ODE integrationsingle system
OpsSplit [5]incompressible or compressible Navier-Stokesthe nonlinear operatorfixed linear operators and neural-ODE integrationsingle system
Origo [6]multi-PDE pre-trainingboth split operatorsone split-step skeletonyes
DISCO [7]multi-PDE pre-trainingan operator generated by a hypernetworkgeneric time integrationyes
HotPOTmulti-PDE pre-traininga different intermediate quantity for each familya different parameter-free update for each familyno, heterogeneous by design

Three results in the paper follow from this difference:

  • Heterogeneity produces the gain. The Direct-state ablation of Table 3 replaces the family-specific interfaces by one next-state interface. The architecture, the stored parameters (97,902,720) and the training protocol stay the same. The error on NS ν=1e-5 / NS ν=1e-3 / CNS / DR rises from 0.0315 / 0.00325 / 0.00671 / 0.0150 to 0.0525 / 0.00625 / 0.0104 / 0.122.
  • The experts shape the shared encoder during pre-training. The encoder is trained through all five updates at once. When the shared computation is moved after the family-specific coupling (Late-shared evolution, Table 2), the mean error rises from 0.0101 to 0.0286.
  • Conservation and positivity hold by construction. The CNS update preserves total mass, momentum and energy and keeps density and pressure positive, and the SWE update preserves mass. Relative to the Direct-state ablation, the measured mass drift over full rollouts falls from 2.04e-4 to 7.63e-8 on CNS and from 9.31e-5 to 1.19e-9 on SWE (Table 12).

The backbone blocks come from DPOT [1] and MoE-POT [2] on purpose. HotPOT is compared with these two models, so we keep their blocks and change only what happens after the history is encoded. The gain over these baselines therefore comes from the heterogeneous experts and not from a stronger backbone.

A shared encoder with one head per task is standard multi-task learning. The “experts” are chosen by dataset name, with no router, so the MoE framing is too strong. The conservation proofs are correct but very simple.

  • A shared pre-trained part combined with specialised modules is the common paradigm of foundation models. BERT [8] is used with task-specific heads, pre-trained vision backbones with detection and segmentation decoders [9, 10], and Segment Anything [11] pairs one image encoder with a prompt encoder and a mask decoder. HotPOT follows this paradigm for PDEs. What is new is the specialised module itself. It is not an output head. It is a complete expert with cross-field coupling, evolution blocks, a decoder and a numerical update.
  • Experts are assigned to PDE families and not to datasets, and the selection is not made by hand. The two Navier-Stokes settings share one expert, and so do the three CFDBench geometries. Each input carries its PDE family as part of the problem specification, and the model dispatches it to the corresponding expert automatically, in training and at inference. This differs from a learned router by design: a router has to infer which experts to use, whereas the family is known whenever a PDE is solved. The analysis in the MoE-POT paper [2] shows that its learned gate identifies the source dataset with about 98% accuracy, so it learns the same information.
  • The simplicity of the proofs is a strength. The proofs guarantee that the implemented updates preserve the stated invariants, and the measurements confirm it: the conservation drift falls by four orders of magnitude relative to the Direct-state ablation. A simple construction without parameters is easy to verify and to reuse.
AuthorsOpenReview ↗

Response to Reviewer nYtC (3/8)

W2. The design and the goal of pre-training

W2. The design goes against the goal of pre-training. Each new PDE needs a hand-written update and a new predictor. A pre-trained model should help with new equations, but here the manual work grows with every family. The only new-family test is the wave equation, and it also needs a manual setup.

The goal of pre-training is to learn, from many systems, knowledge that transfers to new conditions and new equations with little or no target data. A module that is specific to each PDE does not conflict with this goal. In HotPOT the knowledge that transfers lives in the shared history encoder, and the experts carry the physics. HotPOT delivers every form of generalisation expected from a pre-trained model, and it is far ahead of MoE-POT at the same activated scale:

  • Zero-shot transfer (Section 5.3, Table 4). On CNS at an unseen (M, η) = (1, 1e-2), HotPOT reaches 0.189 against 0.423 for MoE-POT-T. On six CFDBench groups with unseen boundary conditions, geometries and fluid properties, its error is 2.5 to 6.7 times lower than MoE-POT-T in every group.
  • Few-shot transfer (Section 5.3, Table 5). With 16 and 64 trajectories on NS ν=1e-4, the error of HotPOT is more than one third lower than that of MoE-POT-T. On CNS (M, η) = (1, 1e-2) it is more than 45% lower at every budget.
  • Transfer to new PDE families. Two families that are absent from pre-training are added by reusing the pre-trained encoder with one new predictor each: the wave equation (Section 5.4, Table 11) and a reacting flow with 15 coupled fields (simulated combustion from RealPDEBench [15], new in the revision).

A new PDE does not need a hand-written update. In our transfer experiments on new PDE families, the wave equation and combustion, we used the generic update in both cases. The predictor outputs the increment of the state and the update adds it, so nothing in it is written for the new equation. Each family required only the list of its observed variables and a new predictor on top of the pre-trained encoder. Even in this setting, the error of HotPOT is 12 to 40% lower than that of fully fine-tuned MoE-POT-T on the wave equation and 17 to 42% lower on combustion, on held-out operating conditions. This shows that, with the general representation learned by the pre-trained encoder, HotPOT is competitive even without any design that is specific to the PDE.

A numerical expert that is designed for the physics of a new family can improve the accuracy further. On the pre-training families, the numerical updates lower the error by 32 to 82% on NS, CNS and DR (W5 / Q2). In future work we also plan to explore the automatic selection and composition of numerical updates, which builds the update of each PDE from a library of existing numerical methods. This will reduce the cost of manual design and improve scalability.

Revision location: Appendix A.15 (combustion experiment) and Section 6 (automatic composition as future work).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General response

We thank the Area Chair and the Reviewers for their careful reading. The main concerns are the novelty of the design, the source of the gain, the fairness and robustness of the comparison, the cost, the generalisation to new and coupled systems, and reproducibility. We have addressed every one of them, by clarification or by new experiments. The table gives an overview, and the detailed answers are in the response to each reviewer. All new results are in the revised PDF, with changes in blue.

ReviewerConcernOur responseStatus
nYtCNovelty (W1)Earlier neural-numerical methods use one template for one system or for every system. In HotPOT each PDE family has its own predicted quantity and its own update inside one pre-trained model. A shared pre-trained encoder with specialised modules is the common paradigm of foundation models. Experts are dispatched automatically.Addressed by clarification
nYtCThe design goes against the goal of pre-training (W2, Q3)The pre-trained encoder transfers. Zero-shot error is 2.2 to 6.7 times lower than MoE-POT-T, and two families absent from pre-training are added with a generic update that needs no term of the equation.Addressed by clarification and new experiments
nYtCScale, single seed, no error bars (W3, Q4)The benchmark is the standard setting of DPOT and MoE-POT. We added paired bootstrap intervals and two further seeds. Every seed is at least 29% below MoE-POT-T on every setting.Addressed by clarification and new experiments
nYtCCost comparison (W4, Q1)One task stores 16.8 to 17.2M parameters against 30.2M for MoE-POT-T. Total inference time is 3297 ms against 3331 ms, and memory is on par.Addressed by new measurements
nYtCSource of the gain, missing baseline (W5, Q2)The requested baseline is exactly the Direct-state ablation of Table 3. Over three seeds, the numerical update lowers the error by 32 to 82% on NS, CNS and DR at the same stored parameters.Already in the paper, confirmed by new seeds
nYtCBaselines and learning rates (W6, Q4)We pre-trained scOT, the Poseidon architecture, on the same six datasets, and HotPOT-T is better in all twelve comparisons. Few-shot learning rates are selected for both models from the same candidates.Addressed by new experiments
nSbbPredefined systems and coupled multi-physics (W1, Q1)A new family needs no new update. On simulated combustion with 15 coupled fields, the error is 17 to 42% lower than MoE-POT-T on held-out operating conditions.Addressed by a new experiment
nSbbStorage and compute as families are added (W2, Q3)Neither grows with the number of families. A task uses the shared encoder and one expert.Addressed by new measurements
nSbbRequired priors and unknown PDE type (Q2)Prior knowledge is optional. Known structural properties, such as conservation laws, are built into the update. A new or unknown system uses the generic update.Addressed by clarification
7FZ8Reproducibility (W1, Q1)Code, checkpoints, test indices and evaluation commands are released.Addressed by release
7FZ8Comparability of the baselines (W2, Q2)The baseline rows come from MoE-POT Table 1. We re-evaluated the released DPOT-T and MoE-POT-T checkpoints through our pipeline, and the deviation from the cited values is below 0.8%.Addressed by clarification and new experiments
7FZ8Matched computational budgets (Q3)Pre-training takes 26.1 GPU-hours against 34.5 for MoE-POT-T. Inference time and memory are on par.Addressed by new measurements

New material added during the discussion period:

  • Code and checkpoints. The anonymous code is at https://anonymous.4open.science/r/HotPOT-BA73/ and the anonymous checkpoints are at https://anonymous-hf.com/a/ryou1c8aizn3/. The 14 checkpoints behind the 24 HotPOT entries of Table 1 are released with SHA-256 sums, test-row indices, expected errors and the exact evaluation command of each entry.
  • Statistical evidence. Paired bootstrap intervals over test trajectories against the released MoE-POT-T and DPOT-T checkpoints, and three training seeds of the full model and of the Direct-state ablation.
  • A new baseline. scOT, the Poseidon architecture, pre-trained and fine-tuned by us on the same six datasets under the same protocol.
  • Measured cost. Training GPU-hours, total inference time and peak memory on the same hardware, and the storage needed for one task. We also release single-expert checkpoints whose predictions are bit-identical to those of the full models.
  • A coupled reacting flow. Simulated NH3/CH4 combustion with 15 coupled fields from RealPDEBench, evaluated on held-out operating conditions.
  • A wrong numerical update. On DR, an update that imposes a false conservation law raises the error after fine-tuning from 0.00163 to 0.412.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.