acceptodds
Under review as a conference paper at ICLR 2027

Warm-starting PDE solvers with any-dimensional machine learning

Abstract

Any-dimensional machine learning models, such as graph neural networks (GNNs), can be naturally trained and evaluated on inputs of different sizes and dimensions. Inspired by the GNN transferability literature, we show mathematical conditions under which a partial differential equation (PDE) learning-based solver can be trained in small dimensions and directly applied to solve a higher dimensional PDE in a zero-shot fashion. These conditions are based on symmetries in both the partial differential equation and the initial data. When the equations satisfy the symmetries but the data does not, which is the case for many PDEs arising from physics, we show that our theory gives a principled way of warm-starting low-dimensional PDE solvers for higher dimensional PDEs. We apply this method on the heat equation, Burgers' equation, and the compressible Navier–Stokes equations, improving the performance in both zero-shot and typical training regimes on high dimensional data. For example, we train a surrogate model on 2D Navier–Stokes data and achieve better results on 3D test data than a baseline surrogate model trained on 3D data, while only using 12% of the flops and 20% of the total data size.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
any-dimensional, pde, symmetries

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_X8oKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces warm-starting PDE solvers with any-dimensional machine learning—a mathematical framework for expressing PDEs across different dimensions. Under this framework, the authors describe explicit conditions under which an n+1n+1 dimensional PDE can be learned from an nn-dimensional machine learning model. For general higher-dimensional initial data that do not satisfy exact-lifting conditions, the paper demonstrates that lower-dimensional pretraining still provides a useful warm start.

Critical strengths

The paper is well written and introduces several abstract mathematical concepts progressively, making it accessible to a broader audience. The overall structure is logical and the narrative is engaging. Furthermore, the method provides a clear mathematical rationale for how parameters should transfer across dimensions.

The proposed framework addresses an important computational challenge: training 3D simulations is computationally demanding, so leveraging easier-to-obtain 2D counterparts is a promising direction. The paper uses numerical experiments (e.g., Navier-Stokes results) to show savings in both training budget and data storage.

Critical weaknesses
  1. Exact compatibility is not directly validated experimentally. Although the paper illustrates why lifting should be exact under the stated constraints, it does not report an experiment verifying this property. While mathematically straightforward, an empirical check would confirm that filter conversion and the complete neural network preserve the claimed compatibility. Given its central role, this inexpensive validation would strengthen the link between theory and implementation.

  2. Benefits are demonstrated on a single architecture family. All experiments rely on a geometric U-Net implemented in ginjax, leaving it unclear whether the observed improvements extend to other neural network or neural operator architectures. Testing lower-dimensional pretraining on additional models—including architectures with and without explicitly enforced any-dimensional compatibility—would provide stronger evidence for the approach's generality.

Questions for authors

Do the warm-start benefits extend beyond ginjax to other architectures, including those with and without explicitly enforced any-dimensional compatibility?

2 comments

W1 (Exact compatibility): Thank you for this suggestion. We have tested this empirically by randomly generating 2D input scalar and vector input data from a normal distribution and embedding this data in 3D using the phi embedding. We then initialized a random 2D equivariant UNet, and used the Nepo warm-start coefficient conversion to convert this to a 3D model. With these we can test the compatibility condition by mapping the 2D input through the 2D model, then embedding it in 3D, and compare it to mapping the embedded 3D data through the 3D model. When we do this, we get a mean-squared error of 9.654e-05. For comparison, a randomly initialized 3D model gets an error of 0.146.

The reason it is 9.654e-05 and not 0 is that small numerical errors in compatibility can build up over multiple layers. However, we used the large UNet (Equiv) model used for the Navier--Stokes experiments, so despite many layers these errors are small.

W2/Q1 (Extensions to other architectures): Two important points prevent any-dimensional machine learning methods from being applied to any general neural network. First, as we explain in the paper, some form of symmetries are necessary to use the framework of representational stability that allows us to perform any-dimensional machine learning. Second, even after choosing a model that implements suitable symmetries, the conditions on the model architecture must be proven for each model individually, as we do in Theorem 4.1. With further significant work, more equivariant models could be adapted to be any-dimensional, but that is outside the scope of this work.

Reviewer_X8oKOpenReview ↗

Thank you

Thank you for the response and the additional compatibility experiment, which helps connect the theoretical result to the implementation. I also appreciate the clarification regarding the architectural requirements and scope. I will maintain my score of 3.

Reviewer_tmhAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when a learned PDE solver trained in n spatial dimensions can be transferred to n+1 dimensions. The authors consider evolution equations whose right-hand side is an O(n)-invariant function of u and its derivatives (e.g., the heat equation) and show that a solution in n dimensions, extended as a constant along a new coordinate, remains a solution in n+1 dimensions (Prop. 3.3). They then give a condition on convolution filters (Thm. 4.1) under which the geometric CNNs of Gregory et al. (2025) commute with this embedding, and propose the "Nepo" warm-start, which chooses the (n+1)-dimensional filter coefficients by matching random n-dimensional slices under an iid-pixel assumption (Def. 4.5, Prop. 4.2), giving closed-form coefficients (Table 2). Experiments on the heat equation (1D to 2D), Burgers' equation (2D to 3D), and compressible Navier-Stokes (2D to 3D) compare the warm-started model against the same architecture trained from scratch on varying amounts of higher-dimensional data. The headline result is that the zero-shot Navier-Stokes model (22.8% relative error) beats a 3D baseline trained on 8 trajectories (27.2%) at a fraction of the compute.

Critical strengths

The core observation is simple and sensible: for Euclidean-invariant PDEs, a low-dimensional solution is a degenerate higher-dimensional one, and an equivariant CNN can be lifted so that it respects this exactly. The resulting warm-start is cheap (closed-form coefficients in Table 2), and on Navier-Stokes it gives a real improvement over training from scratch with little 3D data (Table 1).

Critical weaknesses
  1. The motivation and intended use case are unclear. The paper is extremely formal mathematically (jet bundles, prolongations, representation stability) but written fairly casually, and I found it hard to follow. More importantly, after working through the formalism I'm left unsure when I'm supposed to use this. Is the goal to solve PDEs in genuinely high dimension using only low-dimensional data? Or is it that 3D data is expensive to generate, so one pre-trains on 2D instead? These are quite different settings with different baselines and different success criteria, and the paper doesn't commit to either. The exact transfer result only covers data that is constant along the new coordinate (as the authors acknowledge in Sec. 6), so the practical value of the method rests entirely on the empirical warm-start, and the formalism of Secs. 2 and 3 does little to explain why that warm-start should work on generic data.
  2. The numerical experiments are fairly weak, and this is mostly a theoretical paper. The only comparison is against the same architecture trained from scratch; there is no comparison to alternatives like embedding the low-dimensional data in 3D (as in Walrus, which the authors cite) or other transfer strategies. The data regimes are small (e.g., 8 Burgers trajectories), several of the larger-data settings time out (Table 1), and the coefficient ablation (App. G.3) is run only on the heat equation, which is the case where the theory is closest to exact. Given that the practical contribution is the warm-start, I would expect the experiments to carry more weight.
Questions for authors
  1. (Clarification) What is the intended use case? Specifically, is the target regime (a) high-dimensional PDEs where only low-dimensional data is available, or (b) 3D problems where 2D data is cheaper to generate? A clear answer, with an example application, would substantially change how I read the contribution.
  2. (Clarification) How much of the formalism in Secs. 2 and 3 is needed for the algorithm? As I understand it, the method reduces to lifting filters so that convolution commutes with constant extension (Thm. 4.1) and fixing the remaining free parameters by least squares (Prop. 4.2). If that's right, could the paper be presented with much less machinery?
  3. (Experiments) Can the authors compare against a baseline that embeds the 2D training data in 3D, as in Walrus, and against a warm-start without the compatibility constraint? This would show whether the gain comes from the specific Nepo coefficients or simply from pre-training on related data.
3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

(Motivation): About the experiments:

The primary use case we evaluate is (b): 3D PDE problems for which 2D simulation data is substantially cheaper to generate, and 3D training data or computational resources are limited. For example, in fluid dynamics, we pretrain a solver on 2D simulations, lift its learned parameters to a 3D model, and fine-tune using a small number of 3D training trajectories. Our results show that this is a very effective warm-start strategy, improving accuracy compared with training from scratch with the same amount of 3D data.

The cost of lower-dimensional pretraining is a small fraction of the cost of 3D training, and the pretraining compute is included in our reported training costs. Therefore our approach of initializing a solver with a model pretrained in lower dimensions is not only mathematically principled but also cost effective. Although the mathematical framework is formulated for general spatial dimensions, our empirical evaluation focuses on transfer up to 3D. We will clarify this in the introduction.

About the theory:

We agree that the exact transfer result only holds for inputs that are constant along the added coordinate, and does not guarantee improved performance on generic 3D data. Sections 2 and 3 establish conditions under which PDEs and their solution operators are consistent across dimensions. This gives a justification for the transfer construction: the lifted model should preserve its learned behavior on embedded lower-dimensional inputs. Of course this does not determine its behavior on other inputs. The effectiveness of the resulting initialization on generic 3D data is then demonstrated empirically. The transferability results from Levin et al 2025 may allow one to quantify the generalization error for this case in terms of how far the 3D data is from the embedding of the 2D data, though this analysis is outside the scope for this paper.

(Motivation for Secs. 2,3): It is true that Algorithm 1 needs only the compatibility condition of Theorem 4.1, under which convolution commutes with constant extension, and the least-squares choice of the remaining coefficients in Proposition 4.2. Sections 2 and 3 are not needed to run the algorithm, but they explain why this is the right constraint. For PDEs written with O(n)-invariant differential operators, the true solution operator commutes with constant extension (Proposition 3.3), and Theorem 4.1 is the discrete analogue for the network. We added a comment in the introduction saying that the algorithm itself can be defined without the theory.

AuthorsOpenReview ↗

Rebuttal 2/2

(Numerical Experiments): Thank you for this feedback about the experiments, we have made great strides to improve this section of our paper. First, we have added baseline models including an FNO, a non-equivariant UNet, and the persistence baseline. We have updated the relevant figures and tables in the main body and Appendix G.5. However, the Navier–Stokes experiments are still running, so we will update those when they finish, likely in the next day or so. For Burgers’ equation, the non-equivariant UNet does quite well as the number of tuning points increases.

PDEModels014832128
HeatUNet (Equiv)105.3%21.0%20.8%19.8%3.3%0.5%
UNet169.3%26.3%23.6%23.2%16.8%15.1%
FNO511.5%92.8%92.0%83.2%69.0%59.6%
Persistence470.2%470.2%470.2%470.2%470.2%470.2%
Nepo UNet (Equiv)5.7%2.2%2.2%2.1%0.7%0.5%
BurgersUNet (Equiv)105.3%2.5%1.8%1.6%Time outTime out
UNet431.2%9.4%3.0%1.3%Time outTime out
FNO131.9%3.7%2.9%2.7%Time outTime out
Persistence4.3%4.3%4.3%4.3%Time outTime out
Nepo UNet (Equiv)12.8%1.8%1.6%1.4%Time outTime out

Next, we have added additional experiments with Navier–Stokes Mach 1.0 data, and we have further tested model stability over longer rollouts. For rollouts we autoregressively iterate the model over the full possible 17 steps, then take the average.

ErrorModels014832
One-stepUNet (Equiv)576.7%61.9%37.4%30.6%7.7%
Nepo UNet (Equiv)36.3%32.8%22.8%15.2%6.7%
RolloutUNet (Equiv)3.5e+06%387.1%77.1%526.9%350.2%
Nepo UNet (Equiv)122.1%91.3%63.6%54.8%38.4%

The results for Burgers’ equation averaged over 10 rollout steps are shown below:

PDEModels0148
BurgersUNet (Equiv)100.4%15.8%11.3%10.2%
UNet535.1%31.9%10.3%5.2%
FNO128.4%19.3%15.2%14.0%
Persistence25.8%25.8%25.8%25.8%
Nepo UNet (Equiv)372.9%13.6%7.8%6.4%

See Appendix G.5 for Burgers’ equation rollouts and G.6 for Navier–Stokes Mach 1.0 data and longer rollouts.

Finally, we added a low-resolution 3D pre-training baseline, Embed (Appendix G.6). Embed uses the same lower-dimensional data as Nepo, but pre-trains the higher-dimensional network directly:

  • Each initial condition is lifted to n+1 dimensions by making it constant along a random lattice direction.
  • The target is the exact (n+1)-dimensional heat solution.
  • Pre-training is lower resolution, it uses side length 16 instead of 64, which keeps its cost comparable to Nepo's.

Embed is then fine-tuned like Nepo, with learning rates tuned independently on the validation set.

From 2D to 3D, Nepo is 3.3 to 6.4 times more accurate than Embed for every number of fine-tuning points (e.g., 1.2% vs. 4.9% with no 3D data), while using equal or less total compute. From 1D to 2D, Embed is more accurate with no fine-tuning data (2.4% vs. 5.7%) but uses 2.8× Nepo's compute. The two are comparable with 1–32 points, and Nepo is more accurate with 128.

AuthorsOpenReview ↗

Additional Navier–Stokes Mach 0.1 Results

Here are the additional results with the Navier–Stokes equations with Mach 0.1. We present all the results in tables (also updated in the paper with figures), then discuss them. First the one step relative error:

PDEModels014832
N-SUNet (Equiv)1,565.7%97.0%56.3%27.2%5.2%
UNet318.7%130.0%99.5%64.5%38.1%
FNO12018.8%312.2%561.2%132.4%26.2%
Persistence33.4%33.4%33.4%33.4%33.4%
Nepo UNet (Equiv)22.8%28.9%17.8%14.3%3.8%

Next the average error over an autoregressive 10 step rollout.

PDEModels014832
N-S Mach 0.1UNet (Equiv)8888.8%7916.4%100.6%82.5%431.8%
UNet295.4%119.6%111.2%73.9%45.5%
FNO76227.0%6279373.5%3590454.0%--
Persistence38.2%38.2%38.2%38.2%38.2%
Nepo UNet (Equiv)57.1%56.0%35.4%31.5%10.9%

Finally, we have updated the estimated flops count for each model using a more accurate method in the JAX library that is able to directly count the flops of the compiled program. Observing the plots indexed by flops it is notable that the non-equivariant UNet and FNO are significantly faster. A decent portion of this difference comes down to the custom implementation used in the UNet (Equiv) model. With additional engineering effort these could likely come down and be more in line with the other models. The flops comparison is most relevant for comparing UNet (Equiv) and Nepo UNet (Equiv).

PDEModelsn=0n=1n=4n=8n=32n=128
HeatUNet (Equiv)09.821e-013.928e+007.856e+003.142e+011.257e+02
UNet01.840e+007.362e+001.472e+015.890e+012.356e+02
FNO02.467e+019.868e+011.973e+027.894e+023.157e+03
Persistence000000
Nepo UNet (Equiv)9.027e+001.001e+011.295e+011.688e+014.045e+011.347e+02
BurgersUNet (Equiv)06.002e+062.401e+074.802e+07--
UNet01.098e+064.397e+068.791e+06--
FNO05.017e+052.006e+064.013e+06--
Persistence0000--
Nepo UNet (Equiv)2.930e+056.295e+062.430e+074.831e+07--
N-SUNet (Equiv)03.720e+061.488e+072.976e+071.190e+08-
UNet03.954e+051.581e+063.163e+061.265e+07-
FNO01.751e+057.006e+051.401e+065.605e+06-
Persistence00000-
Nepo UNet (Equiv)3.530e+067.250e+061.841e+073.329e+071.225e+08-

To summarize these results, the Navier–Stokes experiments are the hardest experiments, and they once again prove the benefit of our Nepo warm-start method. The Nepo method is the most accurate, except in some cases where the Persistence baseline does better. It is likely that if we learned and tested on larger time steps, the learned models would provide more improvement over the persistence baseline. The FNO method can be accurate and fast for one step, but it has a tendency to blow up over longer rollouts, and more fine-tuning data does not help with this problem.

Reviewer_gcd7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a neural PDE surrogate trained in a lower spatial dimension can be transferred to a higher-dimensional problem. It develops symmetry-based conditions for exact transfer and introduces a weight-transfer method called Nepo Warm-start for more general settings where those conditions do not hold.

The authors evaluate the approach on the heat equation, Burgers’ equation, and compressible Navier–Stokes. They report that lower-dimensional pretraining improves prediction accuracy when higher-dimensional training data is limited. In one selected Navier–Stokes comparison, a model trained only on 2D data achieves 22.8% error on 3D data, versus 27.2% for a baseline trained on eight 3D trajectories, using about 12% of the estimated training FLOPs.

The main limitation is that the experiments measure short-horizon prediction rather than sustained solver rollouts. Also, the stated general tensor-compatibility theorem appears to need correction or additional assumptions, as described in the review.

Critical strengths

The paper tackles an important and well-motivated problem: transferring PDE surrogates from lower to higher spatial dimensions, where training data and compute can become expensive. Its formulation usefully distinguishes symmetry of the PDE from the stronger conditions on initial data needed for exact dimensional transfer.

The proposed filter-transfer procedure is concrete, and the controlled experiments show promising low-data improvements. For example, in Table 1, the warm-started model achieves 14.3% error on 3D Navier–Stokes with eight 3D training trajectories, compared with 27.2% for the same-architecture baseline. The zero-shot 2D-trained model also obtains 22.8% error in that comparison. These results suggest that lower-dimensional pretraining can be useful even when exact transfer assumptions do not hold.

Critical weaknesses

Theorem 4.1 appears incorrect for the general tensor outputs it claims to cover. Consider scalar input and a rank-two output, with a center-only filter equal to the identity matrix in each dimension: I₂ in 2D and I₃ in 3D. Both filters are invariant under the stated symmetry group, and the projected filter condition in Eq. (18) holds. But for a constant scalar input, applying the 3D filter gives I₃, whereas applying the 2D filter and then embedding the output gives diag(1, 1, 0). Thus the full outputs differ, contrary to the commuting diagram in Eq. (17). The proof in Appendix E.2 appears to overlook the newly introduced output components. This issue affects the general theoretical claim; it does not by itself invalidate the experiments. The authors should clarify whether their actual scalar/vector architecture avoids this case and prove an appropriately restricted result.

The empirical evidence is narrower than the broader PDE-solver framing. Burgers is evaluated on one-step predictions, and Navier–Stokes uses four input steps to predict one subsequent step. The paper does not establish autoregressive stability, long-horizon accuracy, or conservation behavior. Further, the Copy/Zeros transfer ablation is limited to the heat equation and reuses learning rates selected for the proposed method. The reported results therefore do not yet establish that the transfer rule is the best explanation for the gains on nonlinear, vector-valued problems.

Questions for authors

How does Theorem 4.1 address the identity-filter example above? What additional conditions would ensure that newly introduced output tensor components vanish, and do those conditions hold for every layer used in the experiments?

Could the authors specify the existence, uniqueness, domain, and boundary-condition assumptions behind Proposition 3.3? Constructing a lifted solution does not alone establish equality with a selected solution operator when uniqueness has not been stated.

How does the method perform in autoregressive rollouts, particularly on Navier–Stokes? Error versus rollout horizon and relevant physical diagnostics would help determine whether the one-step gains translate into a useful solver.

Does the proposed transfer rule outperform independently tuned alternative initializations on Burgers and Navier–Stokes, including a simple low-resolution 3D pretraining baseline? How do the methods compare under equal total compute?

Please clarify the accounting behind the headline compute and data savings: are hyperparameter searches, validation, and data generation included? In particular, the Navier–Stokes validation set uses 3D trajectories. The reported training-FLOP ratio itself is consistent with approximately 12% for the selected comparison.

2 comments

(Theorem 4.1): Thank you for identifying this issue, we indeed had an error in our proof. We have found and proven the correct conditions for compatibility. Using 2D to 3D as an example, every pixel in the 2D filter embedded with the zero-pad tensor embedding must agree with the sum of the 3D filter along the additional dimension. However, they only have to agree for the first 2 components for the tensor axes that match the input image, while they have to agree along all 3 components for the tensor axes that will appear in the output. See the updated theorem and proof in the paper for full details.

This update changes the constraints in the case of scalar inputs and 2-tensor outputs, which is the case of your counterexample. However, we do not use this case in our experiments, and fortunately this update does not change the coefficient transfer used in the Nepo warm-start for any of our experiments.

(Autoregressive rollout stability): Thank you for this suggestion, we agree that autoregressive rollout stability is an important part for this type of PDE emulators. We tested the rollout stability with new experiments with Navier-Stokes Mach 1.0 data. For rollouts we autoregressively iterate the model over the full possible 17 steps, then take the average.

ErrorModels014832
One-stepUNet (Equiv)576.7%61.9%37.4%30.6%7.7%
Nepo UNet (Equiv)36.3%32.8%22.8%15.2%6.7%
RolloutUNet (Equiv)3.5e+06%387.1%77.1%526.9%350.2%
Nepo UNet (Equiv)122.1%91.3%63.6%54.8%38.4%

We also tested the Burgers’ equations models rolling out over 10 steps, shown below:

PDEModels0148
BurgersUNet (Equiv)100.4%15.8%11.3%10.2%
UNet535.1%31.9%10.3%5.2%
FNO128.4%19.3%15.2%14.0%
Persistence25.8%25.8%25.8%25.8%
Nepo UNet (Equiv)372.9%13.6%7.8%6.4%

(PDE conditions): Thank you for pointing this out. For Proposition 3.3, we now state as an assumption that the PDE Fn=0F_n=0 is well-posed (i.e. the solution exists, is unique, and depends continuously on the initial data u0u_0). This is assumed throughout the text.

(Low resolution pre-training): Thank you for this suggestion. We added a low-resolution 3D pre-training baseline, Embed (Appendix G.6). Embed uses the same lower-dimensional data as Nepo, but pre-trains the higher-dimensional network directly:

  • Each initial condition is lifted to n+1 dimensions by making it constant along a random lattice direction.

  • The target is the exact (n+1)-dimensional heat solution.

  • Pre-training is lower resolution, it uses side length 16 instead of 64, which keeps its cost comparable to Nepo's.

Embed is then fine-tuned like Nepo, with learning rates tuned independently on the validation set.

From 2D to 3D, Nepo is 3.3 to 6.4 times more accurate than Embed for every number of fine-tuning points (e.g., 1.2% vs. 4.9% with no 3D data), while using equal or less total compute. From 1D to 2D, Embed is more accurate with no fine-tuning data (2.4% vs. 5.7%) but uses 2.8× Nepo's compute. The two are comparable with 1–32 points, and Nepo is more accurate with 128.

All error plots are indexed by total training FLOPs, including pre-training and conversion into a higher dimensional solver. Nepo's pre-training is a small part of the total: 1.3% for heat 2D to 3D with 128 points, and 2.8% for Navier-Stokes with 32 trajectories. We ran the Embed comparison on the heat equation, where independent learning-rate sweeps in 3D are affordable. We have not run it for Burgers or Navier-Stokes since we had limited GPU hours available during the rebuttal period.

(Flops accounting): The flops calculation is the total number of flops to train each model for 50 epochs on the higher dimensional training data. For pre-trained models, this includes the pre-training, model transfer, and fine-tuning time. This does not include evaluating the performance on a validation set or performing hyperparameter tuning, but these would not change the gigaflops usage of each model relative to other models. We have updated the discussion to include this explanation in the paper.

AuthorsOpenReview ↗

Additional Navier–Stokes Mach 0.1 Results

Here are the additional results with the Navier–Stokes equations with Mach 0.1. We present all the results in tables (also updated in the paper with figures), then discuss them. First the one step relative error:

PDEModels014832
N-SUNet (Equiv)1,565.7%97.0%56.3%27.2%5.2%
UNet318.7%130.0%99.5%64.5%38.1%
FNO12018.8%312.2%561.2%132.4%26.2%
Persistence33.4%33.4%33.4%33.4%33.4%
Nepo UNet (Equiv)22.8%28.9%17.8%14.3%3.8%

Next the average error over an autoregressive 10 step rollout.

PDEModels014832
N-S Mach 0.1UNet (Equiv)8888.8%7916.4%100.6%82.5%431.8%
UNet295.4%119.6%111.2%73.9%45.5%
FNO76227.0%6279373.5%3590454.0%--
Persistence38.2%38.2%38.2%38.2%38.2%
Nepo UNet (Equiv)57.1%56.0%35.4%31.5%10.9%

Finally, we have updated the estimated flops count for each model using a more accurate method in the JAX library that is able to directly count the flops of the compiled program. Observing the plots indexed by flops it is notable that the non-equivariant UNet and FNO are significantly faster. A decent portion of this difference comes down to the custom implementation used in the UNet (Equiv) model. With additional engineering effort these could likely come down and be more in line with the other models. The flops comparison is most relevant for comparing UNet (Equiv) and Nepo UNet (Equiv).

PDEModelsn=0n=1n=4n=8n=32n=128
HeatUNet (Equiv)09.821e-013.928e+007.856e+003.142e+011.257e+02
UNet01.840e+007.362e+001.472e+015.890e+012.356e+02
FNO02.467e+019.868e+011.973e+027.894e+023.157e+03
Persistence000000
Nepo UNet (Equiv)9.027e+001.001e+011.295e+011.688e+014.045e+011.347e+02
BurgersUNet (Equiv)06.002e+062.401e+074.802e+07--
UNet01.098e+064.397e+068.791e+06--
FNO05.017e+052.006e+064.013e+06--
Persistence0000--
Nepo UNet (Equiv)2.930e+056.295e+062.430e+074.831e+07--
N-SUNet (Equiv)03.720e+061.488e+072.976e+071.190e+08-
UNet03.954e+051.581e+063.163e+061.265e+07-
FNO01.751e+057.006e+051.401e+065.605e+06-
Persistence00000-
Nepo UNet (Equiv)3.530e+067.250e+061.841e+073.329e+071.225e+08-

To summarize these results, the Navier–Stokes experiments are the hardest experiments, and they once again prove the benefit of our Nepo warm-start method. The Nepo method is the most accurate, except in some cases where the Persistence baseline does better. It is likely that if we learned and tested on larger time steps, the learned models would provide more improvement over the persistence baseline. The FNO method can be accurate and fast for one step, but it has a tendency to blow up over longer rollouts, and more fine-tuning data does not help with this problem.

Reviewer_2FouOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the transferability of trained PDE models from nn-dimensional to (n+1)(n+1)-dimensional data. Under the defined compatibility condition, solutions of the nn-dimensional PDE solve the (n+1)(n+1)-dimensional version of the same PDE (after extending them as constants along the new axis). In their transfer experiments, the authors use the Nepo Warm-start algorithm, which transfers the filters of a trained nn-dimensional model to n+1n+1 dimensions. They test their approach on PDEs such as the heat equation, Burgers' equation, and the compressible Navier–Stokes equations.

Critical strengths
  • The problem that the authors tackle is relevant. 3D data are usually expensive to obtain, so transfer from 2D to 3D can be very useful.

  • The authors compare their approach to a natural, controlled baseline (the same model trained from scratch).

  • The paper reports FLOPs, which is very useful in transfer-learning studies.

  • The theoretical part is formal and well-written.

Critical weaknesses

Experiments:

  • The experimental setups for the Burgers' and Navier–Stokes equations, such as initial conditions, time step sizes and time horizons, as well as visualizations of the solution operators, are not described/given. No predicted vs. ground-truth fields are shown.
  • I believe that the experiments are not challenging enough. The heat equation and the Burgers' have smooth and periodic solutions, while the NS experiment is periodic with a low Mach number. Moreover, the NS dataset has 21 time steps, but the model is tested only one step ahead. No autoregressive testing is performed. More challenging experiments should be included. Examples that could be included are the compressible Navier–Stokes data of PDEBench at Mach 1.0, available in both 2D and 3D, and the turbulent radiative layer of The Well, also available in 2D and 3D.
  • It would be interesting to see how the error of the "out-of-plane" velocity component (the one the 2D pretraining does not see) varies with an increasing number of training samples.

Baselines:

  • There are no comparisons to standard baselines such as FNO, standard UNet, and transformer-based methods. It is hard to judge whether the reported errors are competitive with standard models. From the results, I can only conclude that the warm-start is useful in the case of the ginjax model, but I cannot judge whether the warm-start is useful in practice.
  • For the Burgers' and Navier–Stokes experiments, the model predicts one time step ahead. The authors could report the persistence baseline, u^t+1=ut\hat u_{t+1} = u_t, which could help us judge the zero-shot performance. Moreover, the baseline's zero-shot performance only reflects the performance of a randomly initialized network.

Other:

  • Sections 2 and 3 provide justification for inputs that are constant along one axis (i.e., when the inputs are in fact 2D). For general 3D inputs, the theory does not give any guarantee. Remark 3.1 does not really help for large deviations from this subset (i.e., for real 3D inputs). It is unclear to me why the warm-start should help in general 3D regimes.

  • It is unclear to me how the proposed method can be used when the boundary conditions are not symmetric, which is the case in many practical problems (e.g., walls, obstacles, etc.). The authors should clarify whether the warm-start would still help in this setting. The authors should also state how boundary conditions are encoded in the network.

Questions for authors
  1. How do the results change when all models are trained to convergence with a common step budget or early stopping?
  2. What is the error of the persistence baseline for Burgers and Navier–Stokes?
  3. What are the per-channel Navier–Stokes errors for different sample sizes?
  4. Does the approach remain useful with non-periodic boundaries?
4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

W1 (Experimental setups): We used the APEBench package to generate Burgers’ data which generates initial conditions using a truncated Fourier series with 5 modes, then normalizes them to have zero mean and max magnitude 1. APEBench uses a pseudo-spectral Exponential Time Differencing Runge Kutta method which allows it to take time steps of size 1, and each trajectory consists of 50 steps for a total time horizon of 50.

We used the PDEBench package to generate Navier–Stokes data. It generates initial conditions as a sum of sine waves and takes time steps of size 0.05 with a total time horizon of 1. We have added these details in Appendix G.1.

We have included example images of our model prediction vs. ground-truth fields in Appendix G.7.

W2 (Stronger experiments): Thank you for this suggestion, we have added experiments with Navier-Stokes Mach 1.0 data, and we have further tested model stability over longer rollouts. For rollouts we autoregressively iterate the model over the full possible 17 steps, then take the average.

ErrorModels014832
One-stepUNet (Equiv)576.7%61.9%37.4%30.6%7.7%
Nepo UNet (Equiv)36.3%32.8%22.8%15.2%6.7%
RolloutUNet (Equiv)3.5e+06%387.1%77.1%526.9%350.2%
Nepo UNet (Equiv)122.1%91.3%63.6%54.8%38.4%

The results for Burgers’ equation averaged over 10 rollout steps are shown below:

PDEModels0148
BurgersUNet (Equiv)100.4%15.8%11.3%10.2%
UNet535.1%31.9%10.3%5.2%
FNO128.4%19.3%15.2%14.0%
Persistence25.8%25.8%25.8%25.8%
Nepo UNet (Equiv)372.9%13.6%7.8%6.4%

See Appendix G.5 for Burgers’ equation rollouts and G.6 for Navier–Stokes Mach 1.0 data and longer rollouts.

W3/Q3 (Out of plane): We ran this analysis for Burgers’ equation and added the results in appendix G.4. We will also add the results for Navier-Stokes when that finishes as well. Overall the variability between components is low. It is slightly larger in the case of non-equivariant models with fewer training points.

ModelsComponent0148
UNet (Equiv)x105.5%2.5%1.8%1.6%
y105.6%2.5%1.8%1.6%
z104.8%2.5%1.8%1.6%
UNetx397.7%9.3%3.0%1.3%
y322.5%9.2%2.9%1.2%
z420.9%9.9%3.0%1.3%
FNOx162.9%3.8%2.9%2.7%
y135.7%3.8%2.9%2.7%
z125.2%3.7%2.9%2.7%
Nepo UNet (Equiv)x12.8%1.8%1.6%1.4%
y12.7%1.8%1.6%1.4%
z12.8%1.8%1.6%1.4%

W4/5, Q2 (Baselines): Thank you for these excellent suggestions. We have added baseline models including an FNO, a non-equivariant UNet, and the persistence baseline. We have updated the relevant figures and tables in the main body and Appendix G.5. However, the Navier–Stokes experiments are still running, so we will update those when they finish, likely in the next day or so. For Burgers’ equation, the non-equivariant UNet does quite well as the number of tuning points increases.

PDEModels014832128
HeatUNet (Equiv)105.3%21.0%20.8%19.8%3.3%0.5%
UNet169.3%26.3%23.6%23.2%16.8%15.1%
FNO511.5%92.8%92.0%83.2%69.0%59.6%
Persistence470.2%470.2%470.2%470.2%470.2%470.2%
Nepo UNet (Equiv)5.7%2.2%2.2%2.1%0.7%0.5%
BurgersUNet (Equiv)105.3%2.5%1.8%1.6%Time outTime out
UNet431.2%9.4%3.0%1.3%Time outTime out
FNO131.9%3.7%2.9%2.7%Time outTime out
Persistence4.3%4.3%4.3%4.3%Time outTime out
Nepo UNet (Equiv)12.8%1.8%1.6%1.4%Time outTime out

W6 (General 3D inputs): We thank the reviewer for pointing out the discrepancy between exact lifts of 2D inputs versus general 3D inputs; Indeed, the theory rigorously supports transfer of the solution operator for the subset of 3D inputs that arise from a lift. We agree that remark 3.1 is a partial remedy regarding small perturbations (at least in the general case). The main, empirical point our manuscript makes is that enforcing this `agreement’ on the restricted subset of such 3D inputs provides a useful and efficient warm-start, and can act as pre-training for models subsequently tuned to fewer nontrivial true 3D inputs. Our results support the latter interpretation, which, we agree, is not guaranteed by the theory itself; that is in part why we believe the work is interesting.

AuthorsOpenReview ↗

Rebuttal 2/2

W7/Q4 (Boundary conditions): We thank the reviewer for bringing up this point of confusion, and address these issues below:

  • The proposed architecture does not require symmetric boundary conditions (which are indeed often chosen when constructing benchmarking data sets). The predicted field u^(t)\hat{u}(t) only depends on the initial field u(0)u(0), which can be arbitrary in the low-dimensional “pre-training stage”. When the field is lifted to a higher-dimensional space, the rigorous transfer theorem requires that the lifted initial condition is constant along the added coordinate, which is a restrictive assumption.

  • Obstacles can be incorporated if they satisfy the same compatibility condition. For example, a flow around a circular obstacle in a 2D channel can be `lifted’ to a flow around a cylinder in a 3D channel, where the obstacle is also extended along the additional coordinate. We have added this computational example in Appendix G.9, in which we compare Nepo with a baseline for compressible Navier Stokes in that setup.

  • Boundary conditions are not encoded in the network itself. In the supervised low-dimensional training, the boundary conditions enter the loss in the same way that the general field values do (Appendix G).

Q1 (Common training budget): All models are trained with the same budget to convergence. In the setting we defined, all points in the plot correspond to converged models (with a fixed small data set size, fixed common learning rate schedule, and optimal peak learning rate found via a hyperparameter optimization procedure).

AuthorsOpenReview ↗

Additional Navier–Stokes Mach 0.1 results

Here are the additional results with the Navier–Stokes equations with Mach 0.1. We present all the results in tables (also updated in the paper with figures), then discuss them. First the one step relative error:

PDEModels014832
N-SUNet (Equiv)1,565.7%97.0%56.3%27.2%5.2%
UNet318.7%130.0%99.5%64.5%38.1%
FNO12018.8%312.2%561.2%132.4%26.2%
Persistence33.4%33.4%33.4%33.4%33.4%
Nepo UNet (Equiv)22.8%28.9%17.8%14.3%3.8%

Next the average error over an autoregressive 10 step rollout.

PDEModels014832
N-S Mach 0.1UNet (Equiv)8888.8%7916.4%100.6%82.5%431.8%
UNet295.4%119.6%111.2%73.9%45.5%
FNO76227.0%6279373.5%3590454.0%--
Persistence38.2%38.2%38.2%38.2%38.2%
Nepo UNet (Equiv)57.1%56.0%35.4%31.5%10.9%

We can also report the per-component relative error for Navier–Stokes Mach 0.1

PDEModelsComponent014832
N-SUNet (Equiv)x2598.2%216.9%131.0%62.6%12.3%
y2546.3%212.8%127.6%61.6%12.4%
z2550.1%216.3%128.5%61.9%12.4%
N-SUNetx453.3%179.7%93.8%80.5%69.6%
y523.4%227.9%91.3%80.1%70.5%
z568.1%173.7%90.4%80.6%70.0%
N-SFNOx38084.1%786.9%1044.1%261.9%67.0%
y35182.6%751.0%1849.9%273.1%68.1%
z30392.6%731.6%1744.4%226.2%67.4%
N-SNepo UNet (Equiv)x50.8%64.8%34.7%27.0%9.0%
y50.4%63.4%34.3%26.8%9.1%
z50.5%64.1%34.3%27.0%9.1%

Finally, we have updated the estimated flops count for each model using a more accurate method in the JAX library that is able to directly count the flops of the compiled program. Observing the plots indexed by flops it is notable that the non-equivariant UNet and FNO are significantly faster. A decent portion of this difference comes down to the custom implementation used in the UNet (Equiv) model. With additional engineering effort these could likely come down and be more in line with the other models. The flops comparison is most relevant for comparing UNet (Equiv) and Nepo UNet (Equiv).

PDEModelsn=0n=1n=4n=8n=32n=128
HeatUNet (Equiv)09.821e-013.928e+007.856e+003.142e+011.257e+02
UNet01.840e+007.362e+001.472e+015.890e+012.356e+02
FNO02.467e+019.868e+011.973e+027.894e+023.157e+03
Persistence000000
Nepo UNet (Equiv)9.027e+001.001e+011.295e+011.688e+014.045e+011.347e+02
BurgersUNet (Equiv)06.002e+062.401e+074.802e+07--
UNet01.098e+064.397e+068.791e+06--
FNO05.017e+052.006e+064.013e+06--
Persistence0000--
Nepo UNet (Equiv)2.930e+056.295e+062.430e+074.831e+07--
N-SUNet (Equiv)03.720e+061.488e+072.976e+071.190e+08-
UNet03.954e+051.581e+063.163e+061.265e+07-
FNO01.751e+057.006e+051.401e+065.605e+06-
Persistence00000-
Nepo UNet (Equiv)3.530e+067.250e+061.841e+073.329e+071.225e+08-

To summarize these results, the Navier–Stokes experiments are the hardest experiments, and they once again prove the benefit of our Nepo warm-start method. The Nepo method is the most accurate, except in some cases where the Persistence baseline does better. It is likely that if we learned and tested on larger time steps, the learned models would provide more improvement over the persistence baseline. The FNO method can be accurate and fast for one step, but it has a tendency to blow up over longer rollouts, and more fine-tuning data does not help with this problem.

I thank the authors for the detailed response and the additional experiments that addressed some of my concerns. I am willing to raise my score to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.