- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SymOnet, a transformer-based network capable of predicting the PDE solutions at arbitrary space/time query points. The model is pretrained on variety of 2d datasets. It is tested on several downstream tasks, with some of them not seen during the pretraining. The backbone is frozen downstream, while only light adapters are trained. They use physics-based, as well as data-based loss functions for finetuning.
Critical strengths
- The problem the authors tackle is very relevant for the community, as adapting the pretrained PDE models using only physics loss is a core issue.
- The design of the architecture makes it possible to differentiate the model at arbitrary query point.
- A lot of different baselines are tested in their experiments.
- The ablation studies are well-designed.
- The 2d to 3d transfer is very valiuable experiment that shows physics transfer between different number of dimensions
- The appendix is very well-written and detailed.
Critical weaknesses
- Comparisons against the pretrained models, such as Poseidon, DPOT or Walrus in physics-informed settings are not fair ones, given the design of those models. Those models are not designed to be differentiable with respect to query points. Their default design is suitable for data-driven downstream tasks. Finite-difference scheme used in their finetuning may lead to poor estimates of the partial derivatives, particularly the temporal derivatives, which could explain their weak downstream performance. I suggest the authors to report SymOnet finetuning performance with the same finite-difference approach, so that we can separate the effect of the architecture and pre-training from the derivative estimator.
- 8 of the 12 tasks in Table 2 are included in the pre-training. I believe that one of the points of pretrained models in this field is to use them on unseen tasks, so the pre-training comparisons are not very useful. Regarding the out-of-training tasks, the CNS task is an unseen combination of seen parameters of already seen tasks (very similar data distribution). This leaves us with 3 out-of-training tasks. This number of tasks is not enough to judge the performance of a pre-trained model/pipeline. I suggest the authors to run additional experiments, such as some of the multi-scale Navier-Stokes, Euler, MHD or wave experiments from Walrus or Poseidon works. The 2d to 3d transfer is a useful experiment that suggests that the SymOnet may perform well to the new, unseen and generally difficult downstream tasks.
- The previous concerns also lead to the conclusion that the data-free framing is overstated, since in majority of the table 2 experiments, the model has already seen the data during the pretraining.
- The authors set , meaning that only 5 rollouts are applied. It is unclear how much the trajectories are actually moving in this time frame. For SWE in particular, in Table 5, we observe that the frozen model reaches error in zero-shot settings, despite never seeing a single sample from this dataset. The authors should include the trivial persistence baseline and state the L2RE for this baseline for all the OOD experiments. Moreover, the authors should report the errors at longer rollouts, and show how the errors behave in time for the SymOnet against the best-performing baselines.
- It is unclear to me whether the data- and the expression-driven experiments are tested on different time windows. The use of 10 history frames for data-driven models suggests that the testing time-frames for two different regimes differ.
- The authors do not show actual samples and predictions anywhere in the paper or the appendix. It is hard to judge how difficult the tasks are, since no visualization is provided.
Questions for authors
-
For the grid-based baselines, how exactly is the temporal derivative computed?
-
What are the zero-shot errors of the baselines (without any downstream training)?
-
What is the error of the frozen pre-trained SymOnet (no downstream training) on the 8 pre-training tasks from the main table?
-
Could the authors confirm that CNS (1, 1e-8), DR, SWE and CFDBench were excluded from pretraining for SymOnet and other baselines?
-
For SWE, what is the time step between the evaluated frames?