acceptodds
Under review as a conference paper at ICLR 2027

Exponential–Nemytskii Neural Operators: Learning Evolution Operators Across Step Sizes

Abstract

In variable-step operator learning, agreement at observed step sizes does not by itself determine predictions at other temporal resolutions. We propose an operator whose step-size dependence is prescribed by a first-order Duhamel approximation of semilinear dynamics. It combines exact linear propagation with a local nonlinear function of the field and its derivatives, allowing evaluation below the observation interval without supervision densely sampled across step sizes. Under suitable conditions and at fixed spatial resolution, we prove that agreement between models at two step sizes controls their prediction differences throughout a prescribed range. We also give conditions for local identifiability of both components from two step sizes, supporting interpretation of the learned structure. A latent extension applies the same construction to learned fields. We evaluate the base model on two synthetic problems and the extension on two real datasets against several baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
operator learning, semilinear dynamics, Nemytskii operator, polynomial spectral symbol

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GTwCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an innovative evolution operator learning method for nonlinear time dependent partial differential equations (PDEs) of the semi-linear type. The neural operator architecture is deceptively simple, based on a first order approximation to the Duhamel principle (linear part and a local nonlinear part). When the true dynamics is not semilinear, the authors propose a local encoding and decoding to map into a latent space where the dynamics is (hopefully) approximately semi-linear. The training can be performed purely from trajectory data of the PDE. On the theory side, the authors prove two interesting theorems. The first shows that the difference in the action of the model at two distinct time steps controls the difference on a whole interval of times. The second result is an injectivity result again based on two step sizes (with a negative result if only one step is considered). Numerical experiments show better (and sometimes much better) accuracy than baseline methods for learned evolution operators.

Critical strengths

I like the first-order Duhamel parametrization idea. It explicitly exposes the role of the time step τ\tau in the architecture, rather than inserting \tau as another neural network input in a black box way. The paper is quite readable and easily to follow up until Section 2.3

The method has extremely good performance (see Table 3) on some problems, and good but less impressive performance on others. Regardless, the simplicity of the network parametrization should be praised; it gives better performance than much more complicated alternative methods (such as those described in Section 2.4).

Theorem 1 is quite interesting and important for the field of equation learning.

Critical weaknesses

Can Assumption Eqn 13 be verified for specific PDEs of interest, such as those studied in the numerical experiments part? I would like to see more discussion about when the hypotheses of the theorems are satisfied, in order to make sure they are not vacuous empty sets.

I have a hard time understanding the proofs, which are not easy to read. This is not critical, but I am not sure where else to write this.

Questions for authors

What is meant by "persistence" as the last word at the very bottom of page 5?

Table 5: All learned coefficients are of the same order, which seems to contradict the claim in Line 477. Does this suggest that the data could be explained by a different PDE with additional terms defining the differential operator rather than just the Laplacian?

Figure 1: What can explain the oversmoothing as time advances in the bottom row of Figure 1? This is undesirable.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GTwC

We thank the reviewer for the constructive comments. We have added concrete KdV, Swift–Hohenberg and advection–diffusion examples verifying the nonlinear separation assumption. We have provided outlines for the two main proofs and clarified the definition of persistence. We have also expanded the PDE reconstruction analysis to account for both model components and added SEVIR comparisons with a discussion of the oversmoothing. We respond to each point below.

Weaknesses

W1. Verifying Assumption (13).

(Submission: Eq. (13); revision: Eq. (12).)

Yes. The nonlinear separation condition (Eq. (12) of the revision) holds for the reference models of all three synthetic examples: the quadratic transport nonlinearities of KdV and advection–diffusion, and the polynomial reaction term of Swift–Hohenberg. Each satisfies it with an explicit positive separation constant; Appendix A.1.2 gives the proofs and the parameter conditions. So the assumption is satisfied by the PDEs in our experiments.

W2. Readability of the proofs.

We have added a short outline before each of the two main proofs, cross-step stability and identifiability (Appendices A.1.1 and A.2.1), stating the main steps and the lemmas they use.

Questions

Q1. "Persistence" on page 5.

(Submission: page 5; revision: page 6 (Section 2.4).)

Persistence is the baseline that uses the last observed field as the prediction at every future time, u(t0+t)≈u(t0)u(t_0+t)\approx u(t_0). We have defined it at its first mention in Section 2.4.

Q2. Learned coefficients in Table 5.

(Submission: Table 5, Line 477; revision: Table E.2 (Appendix E); Appendices D.1 and D.5.)

We agree that our original comparison, based only on the coefficients of gg, was incomplete for assessing PDE recovery. The learned vector field is Au+f(Φ[u])Au+f(\Phi[u]), and ff can contain linear differential terms as well as nonlinear terms. Therefore, first-order coefficients in gg alone do not establish that the full model represents a different PDE with additional physical transport terms.

We have updated the AD coefficient table (Table E.2, Appendix E) to report, in the same basis, the reference coefficients, the contributions of gg and ff, their sum (the effective coefficients) and E-WSINDy. Appendices D.1 and D.5 describe the procedure: we fit the output of the learned ff on a library of linear and nonlinear differential monomials and add each fitted linear coefficient to the matching coefficient of gg.

The first-order coefficients of gg (−0.22-0.22 and −0.29-0.29) are almost exactly cancelled by ff, giving effective values of −0.00005-0.00005 and 0.00140.0014. The effective diffusion coefficients (0.0490.049 for both directions) and the transport coefficients (−0.99-0.99 and −1.01-1.01) are close to the reference values 0.050.05 and −1-1. The apparent first-order terms in gg are therefore a redistribution between the two components, not additional terms of the learned PDE. The fit residuals and vector-field errors are reported in Appendix D.5.

Q3. Oversmoothing in Figure 1.

(Submission: Figure 1; revision: Figure 3 and Appendix C.4, Figure C.4.)

As the reviewer correctly noticed, the predictions become smoother as time advances. This oversmoothing is a common issue in deterministic precipitation nowcasting, particularly at longer lead times (Ravuri et al., 2021). With squared-error training, the population-optimal prediction is the conditional mean, so averaging possible cell locations and intensities can smooth local features. Autoregressive prediction can carry these errors forward.

To compare with other methods, we have added the full 5–50 minute sequence with KNO and OSG-FNO alongside ENNO in Appendix C.4 (Figure C.4). ENNO preserves the broad storm structure, while small-scale details and localized intensity peaks weaken at longer lead times. KNO and OSG-FNO also lose fine-scale target structure, and KNO shows additional contour deformation.

I thank the authors for addressing my comments.

Reviewer_qWFMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper works on filling a well recognized gap in the neural-operator literature: models are usually trained at a fixed time step and then don't generalize to other step sizes. Towards this effort they propose ENNO, an autoregressive operator for semilinear evolution PDEs on periodic domains. For a single timeset size τ\tau, the operator propagates the field exactly under a learned constant coefficient linear operator (in fourier space; with g(k)g(k) as learnable). It then adds τ⋅f\tau \cdot f where f is pointwise MLP of the propagated field and its spectral derivatives, derived as a first order Duhamel approximation. Because τ\tau enters analytically, the model can be evaluated at step sizes other than those seen in training, including below the observation interval. Training uses one-step losses at two step sizes (The authors prove a cross-step stability bound at fixed resolution). They demonstrate this on a few experiments: kdV, advection-diffusion, radar nowcasting and wildfire spread prediction. On all these cases, ENNO has the best reported metric against the baselines compared (across different timestep choices).

Critical strengths
  • Principled construction that incorporates the timestep size directly into the parameterization of the model
  • Elegant theory to support their methodology (especially motivating single step training is degenerate)
  • Its nice that you can interpret the model by studying the learned polynomial coefficients
  • Good performance below the observation interval τ/2\tau / 2 which other models don't seem to either not support or do worse on.
Critical weaknesses
  • No compute reporting; parameter matching hides compute. The paper reports no wall-clock, FLOPs, memory or hardware, and no per-baseline parameter table
  • Both PDEs are constant-coefficient semilinear equations on periodic boxes with local quadratic nonlinearities. That is exactly the ENNO ansatz, so strong results aren't surprising.
  • The real-world setting isn't supported by the theory (although acknowledged), and the gains are small. In my opinion, the test problems look non-standard. I'd want ENNO evaluated on established suites with rollout baselines, e.g. The Well (Ohana et al., NeurIPS 2024 Datasets & Benchmarks). Its baselines are admittedly not heavily tuned and have since been beaten, but they're a common reference with good rollout evaluation. APEBench or PDEBench periodic tasks (KS, Kolmogorov flow) would also work. As it stands the theory is nice, but the gains on proper problems aren't convincing.
  • Single seed. All reported results come from "a single training seed per configuration"
  • The composed model doesn't satisfy the semigroup property. (not too surprising given the nonlinear component; but I think the influence of this should be studied; see questions)
  • Lacks extensive ablations: Every experiment trains on exactly τ\tau, 2τ2\tau. There's no ablation of the pair, its ratio, more than two step sizes, or even one step size vs two. The theoretical motivation (one step size is non-identifiable) is never tested empirically.
  • Potentially missing discussion of relevant work: the work reminds me a lot of the splitting schemes in the numerical integrator literature (lie and strang splitting etc)
  • Lack of a Figure 1 for clarity: For a method this structured, a Figure 1 would help a lot. It should show a block diagram of one step with the two places τ\tau enters, training at two step sizes vs inference at other steps, and the latent pipeline.
Questions for authors
  • As stated in weaknesses, please report tabulate parameters, FLOPS, wall-clock and memory for each method and task
  • How does a direct step S(kΔt)S(k\Delta t) compare with kk composed steps S(Δt)S(\Delta t) at the same horizon (say at k=2,4,10,100k = 2, 4, 10, 100)?
  • Can you ablate the training step sizes: one vs two, different ratios? Does coefficient recovery degrade with one step size as the lemma predicts?
  • Can you show error-vs-step curves over the full simulated trajectories, against the autoregressive baselines, and confirm long-horizon stability (even on KdV)?
  • What happens without training noise, and at other noise levels?
  • Could you evaluate on standard benchmarks with rollout baselines (e.g. The Well or PDEBench)? This would also be useful since we have established results available on these datasets
  • Can you report results over several training seeds?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qWFM (1/2)

We thank the reviewer for the detailed and constructive review. The main revisions addressing it are:

  1. the parameters, wall-clock time, FLOPs and memory of every method (W1; Appendix B);
  2. the two-dimensional Swift–Hohenberg scenario of APEBench in the main text, and five one-dimensional APEBench scenarios in Appendix F (W2, W3);
  3. five training seeds for every method on the synthetic problems (W4);
  4. direct against composed predictions and ablations of the training step sizes (W5, W6; Appendices C.1, C.2 and D);
  5. a discussion of splitting schemes (W7; Section 2.4) and a method diagram (W8; Figure 1);
  6. error curves over the full test trajectories and results at other noise levels (Q4, Q5).

Weaknesses

W1. Compute reporting (also Q1).

We have added this information to the revision: the number of parameters of every method in the main tables, and the architecture, training settings, wall-clock time, FLOPs, peak memory and hardware of every method and task in Appendix B (Tables B.2–B.6).

W2. Test problems match the ansatz.

We agree that ENNO is designed for semilinear PDEs with constant coefficients. We would argue, however, that this is a wide class of PDEs used in practice, e.g. reaction–diffusion systems, nonlinear waves (KdV, nonlinear Schrödinger), pattern formation (Swift–Hohenberg) and many others (e.g. in PDEBench and APEBench). Following the reviewer's suggestion, we have also tested examples less tailored to ENNO: the second example is now replaced by the Swift–Hohenberg equation of APEBench, which has a cubic nonlinearity, and Appendix F adds five APEBench scenarios, including the two chaotic Kuramoto–Sivashinsky equations (see W3). We also want to emphasize that a key advantage beyond accuracy is that ENNO can predict at step sizes not seen in training, for example at half the observation interval, where KNO and FNO cannot be evaluated (nor can any of the built-in baselines of APEBench).

W3. Real-world setting and standard benchmarks (also Q6).

Standard benchmarks. We followed the suggestion to use APEBench. We changed the second synthetic example in the main text to the two-dimensional Swift–Hohenberg scenario of APEBench, with its default data and the APEBench budgets for the baselines (Section 3.2). We added in Appendix F the five one-dimensional nonlinear scenarios of APEBench against its five built-in baseline methods, with the same data for all methods. ENNO has the lowest error in all five scenarios, by factors of 1.2 to 8.4 relative to the best of these baselines.

Real data. We agree that the theorems do not apply directly to the real data, which use the encoder–decoder. We plan to study the theory in this setting further, and the revision makes an initial effort: Appendix A.1.6 (Corollary A.6) shows that, with a fixed encoder and a Lipschitz decoder, the cross-step stability of the latent model carries over to the decoded predictions. While we do not have a rigorous guarantee in this setting, ENNO can still be used in these problems: it is the most accurate on both real datasets among all tested methods, including standard baselines in these applications.

The advantage of ENNO is not only its accuracy but also its ability to predict at arbitrary step sizes. For example, on SEVIR, its MSE at 50 minutes is 9.6, against 10.3 for KNO (×10−3\times10^{-3}; Table 7), but KNO predicts only at its training step size; against OSG-FNO, which can also predict at arbitrary step sizes, the advantage is larger (9.6 vs 13.7). On WildfireSpreadTS, we corrected an implementation error in the submission and the results are now improved: the mean AUC-PR of ENNO is 0.376 and 0.427 with one and five input days, against 0.291 and 0.310 for the second-best method, which is still a clear margin (Table 8).

W4. Single seed (also Q7).

All results on the two synthetic problems, KdV and Swift–Hohenberg, are now means over five training seeds, with the standard deviation, for every method, including the ablations. The real-data results come from a single training seed (on WildfireSpreadTS, one per fold, over four folds), because of their training cost. This follows common practice in these applications: for example, Earthformer (Gao et al., 2022) on SEVIR and the WildfireSpreadTS benchmark (Gerard et al., 2023) also use a single seed.

W5. Semigroup property (also Q2).

Following the reviewer's suggestion, we compare in Appendix C.2 (Figure C.1) one direct step of size TT with fixed-τ\tau rollout on both synthetic examples, KdV and SH, for TT from 0.1τ0.1\tau to 10τ10\tau. The two coincide for T≤τT\le\tau; between the training steps, the direct step can be better (0.138% against 0.281% at T=1.5τT=1.5\tau on SH); far beyond them, rollout is more accurate (at T=10τT=10\tau, 1.20% against 2.02% on KdV and 0.788% against 8.71% on SH). ENNO is designed for small steps, so horizons far beyond the training steps, such as k=100k=100, should always be predicted by rollout (see Q4).

AuthorsOpenReview ↗

Response to Reviewer qWFM (2/2)

W6. Ablation of the training step sizes (also Q3).

We ran five-seed ablations of the training step sizes on KdV and SH (Appendix C.1) and reconstructed the coefficients of every trained model (Appendix D).

One versus two step sizes. With one step size, coefficient recovery degrades. On KdV, one of the five seeds fails to recover the dispersion coefficient (c3=−0.10c_3=-0.10 instead of −1-1), while all five two-step seeds recover it (−0.998-0.998). On SH, the u2u^2 and u3u^3 coefficients are 1.27 and −1.28-1.28 with one step size, against 0.98 and −0.99-0.99 with two (truth 1 and −1-1). In prediction on SH, one step size is more accurate at multiples of the training step, but less accurate at other step sizes (e.g. 0.315% against 0.161% at t1/2t_{1/2}; Table C.2).

Ratios and more than two step sizes. On KdV, we also trained with {τ,5τ}\lbrace\tau,5\tau\rbrace and {τ,2τ,5τ}\lbrace\tau,2\tau,5\tau\rbrace. Both recover the dispersion coefficient accurately (Table D.3), but have slightly higher errors than {τ,2τ}\lbrace\tau,2\tau\rbrace (e.g. 1.40% and 1.35% against 1.20% at t10t_{10}; Table C.1), so more or more widely separated step sizes do not necessarily help.

W7. Related work on splitting schemes.

Thank you for raising this interesting point, and we agree with the reviewer: the one-step map of ENNO has the form of a Lie splitting, or first-order exponential integrator, Sτ=(I+τF)∘eτA\mathcal{S}_\tau=(I+\tau F)\circ e^{\tau A}. Of course, splitting schemes integrate a known equation, whereas in ENNO, AA and FF are learned, and the step size enters the learned map in a prescribed way, which underlies prediction at unobserved step sizes and our identifiability and stability results. We also found several machine learning works based on this scheme, but they appear to address different problems (e.g. Loya et al. (2025) learn neural ODEs for stiff systems). We added a discussion of these methods to the related work (Section 2.4), and we will compare the two approaches in more detail in future work.

W8. A Figure 1.

We agree and added such a block diagram (Figure 1, Section 2.2).

Questions

Q1. Parameters, FLOPs, wall-clock and memory.

See our response to W1.

Q2. Direct versus composed steps.

See our response to W5.

Q3. Ablation of the training step sizes.

See our response to W6.

Q4. Error over the full trajectories and long-horizon stability.

Following the reviewer's suggestion, we added the error at t100t_{100}, a hundred steps of the observation interval, for every method to the main tables (Tables 3 and 6). We also plotted the error over the full test trajectories, 200 steps on KdV and 191 on SH, for every method and all five seeds, rolled out in steps of the observation interval (Figure 2 for SH at 20% noise; Appendix C.3, Figure C.3, for all settings). ENNO's error grows steadily without blow-up in all settings, including KdV, while one NOMAD seed diverges on KdV. At t100t_{100}, ENNO has the lowest error on KdV and on SH at 20% noise.

Q5. Training noise.

In the synthetic examples, we added experiments without noise and at different noise levels. On KdV (Table C.1), training without noise lowers the error of ENNO at every time, most at t10t_{10} (0.496% against 1.20% with 5% noise) and least at t100t_{100} (3.33% against 3.94%). On SH we report three noise levels (Table C.2): without noise the errors are within one standard deviation of those at 5%. At 5% noise, E-WSINDy is the most accurate method on SH, but at 20% noise its error increases substantially and its accuracy is far below that of ENNO, which is clearly the most accurate of all methods (Table 6).

Q6. Standard benchmarks.

See our response to W3.

Q7. Several training seeds.

See our response to W4.

I thank the authors for the substantial revision. The added compute reporting, multi-seed results, ablations, long-horizon curves and Figure 1 are appreciated and improve the paper.

However, my central concerns aren't fully resolved:

  • Primarily, the test problems seem nonstandard (without adapting to a commonly run suite its hard to judge), and all added problems such as the APEBench scenarios with SH's cubic term and chaotic KS, stay within ENNO's semilinear, constant-coefficient ansatz on periodic domains. The evaluation therefore still does not show that the gains hold on harder established problems outside this class (with established baselines), such as The Well, which I had suggested.
  • Also, from what I can infer the 1D experiments do not use the official default setups: they use different scenario variants, reduce Δt\Delta t from 0.1 to 0.01 in four of five cases, and use a different aggregate metric (these changes are making it much harder to judge).
  • Further, the baselines also omit the transformer-based operators that are now standard choices, which makes it more difficult to place ENNO relative to the current state of the art.
  • As the baselines were run by the authors on self-selected problems rather than taken from a suite with published results, I also find it hard to assess how well tuned they are.

I will keep my score.

We thank the reviewer for the follow-up and for acknowledging the improvements. Below is our response to the remaining points.

Applicability of method (and the test problems). We agree that the synthetic problems are semilinear with constant coefficients on periodic domains. Our intention is to design a method for this class and for problems that can be approximately mapped to this class through the encoder–decoder structure (e.g. the two real-data examples). As mentioned earlier, this class includes a rather large number of important and common mathematical models. We do not claim that ENNO applies to all problems, and we will state this scope more explicitly in the final version.

Benchmark settings. Among the benchmark problems, the two-dimensional Swift–Hohenberg example uses the default APEBench setting. In the one-dimensional problems, we used shorter time steps, to make the examples consistent with the setting we aim to address. A main focus of ENNO is prediction at arbitrary step sizes, in particular at small ones. For example, if a model is trained on data observed every second and asked to predict 0.5 seconds ahead, one would intuitively expect this prediction to be more accurate than the one at one second. However, existing neural operators either cannot predict at a shorter step (FNO and KNO) or are less accurate there than at the training step: on KdV, the error of OSG-FNO is 2.42% at t1/2t_{1/2} against 0.943% at t1t_1 (Table 3). ENNO addresses this problem. We are therefore mainly interested in settings with small time steps. Apart from the step size, we did not change the settings of the benchmark problems, such as the initial and boundary conditions.

Transformer-based operators. We selected the baselines mainly among operators that can predict at arbitrary step sizes (OSG-FNO, TI-DeepONet and NOMAD). Most of them are recent, and they have been compared extensively with existing methods in their original papers. We note that many operator methods cannot predict at arbitrary step sizes; we also included some of them (FNO and KNO) as references at the observation step, but they are not the main baselines of the comparison. If a suitable transformer-based method can address this problem, we would be very happy to compare with it in detail in future work.

Tuning of the baselines. The APEBench baselines use the published configurations and the default training protocol of APEBench (Koehler et al., 2024, Table 5). For the other baselines, the settings on the synthetic examples were selected by ablation, and those on the real data mostly use default values; Appendix B lists every setting and its origin.

As a comment on the last point and the work overall, we would also like to emphasize that outperforming all methods at all times is not our main goal; rather, our main aim is to develop a method that predicts well at arbitrary step sizes, including step sizes shorter than those seen in training, which many existing methods cannot achieve.

Reviewer_cEY1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Exponential-Nemytskii Neural Operators (ENNO), a variable-step operator learning method based on a first-order Duhamel approximation. It combines a spectral linear propagator with a nonlinear Nemytskii operator, and is trained with multiple different step sizes. The authors claim that (1) by encoding dependence on the step size into the model design directly, only two step sizes are needed for stability guarantees across a variety of steps, and (2) training with two step sizes guarantees local identifiability of the linear and nonlinear components. The method is evaluated on two synthetic tasks (Korteweg–de Vries and advection-diffusion) and two real-data applications (radar nowcasting and wildfire prediction) against different families of variable-step methods, though not standard baselines for the real applications.

Critical strengths
  1. The idea is clear and well-motivated: making the step-size dependence part of the construction instead of learning it by data makes it possible to evaluate at any step size without needing dense training data.
  2. The paper is well-written and it’s easy to follow the motivation and model derivation. The theoretical guarantees are proven under reasonable assumptions and the necessity of a second step-size for identifiability is clearly explained.
Critical weaknesses
  1. I think the biggest limitation of this paper is that the central claim, stable variable-step prediction, is only modestly tested. The variable step benefit is only shown at a single time step (t_{1/2}) only on two synthetic problems. The two real data applications are only evaluated at multiples of the training step size. For this claim to be convincing I’d at least expect to see a sweep of the empirical error against a range of step sizes and how it compares with the theoretical bound (perhaps both when trained with only one vs 2+ step sizes), and that ENNO can also be applied on real data on step sizes below the observed interval.
  2. The interpretability of the model components isn’t well-studied. The synthetic data experiments compare the estimated symbol coefficients for the linear operator, but don’t compare the non-linear component against the ground truth. In fact, even the coefficients for the linear component aren’t very convincing: on advection-diffusion the spurious first-order coefficients (~0.05-0.06) are as large as the true coefficient (0.05). On KdV, the spurious coefficients c0 and c1 have an error of 0.06-0.08, which is very different in magnitude to c2 which has an error of 5e-5.
  3. Both of the real data problems (radar nowcasting and wildfire prediction) don’t follow the semilinear structure, requiring the use of the latent field encoder/decoder. To my understanding, this means that not only is the model not interpretable, but Theorem 1 also does not apply. Theorem 3 also requires that K >= K0, but from my calculation K0 ≈ 120 for both real data settings but the K used is only 64 for SEVIR and 32 for Wildfire. So it’s not clear to me how much of the theoretical claims of the paper really apply here. Is the performance actually driven by the exponential-Nemytskii operator or does it just come from the MLP encoder/decoder? Similarly, does variable step size still work in this setting (similar to Point 1 above)?

The idea is presented well, but the experiments don’t support the central claims. The variable-step benefit is only shown on the two synthetic problems, and the real data results don’t really connect with the theory nor do they compare against standard baselines, making them unconvincing. I would raise my score if weaknesses 1-3 are suitably addressed.

Questions for authors
  1. Which of the benchmarks had tuned hyperparameters and which used default values? Please include the chosen hyperparameters in a supplementary table. TI-DeepONet and NOMAD have such large errors on advection-diffusion that it seems possible they may be misconfigured/undertuned.
  2. Why is persistence not included as a baseline on the synthetic tasks?
  3. Why were no standard baselines compared against for the real data tasks, like NowcastNet [1] for radar nowcasting or the models from Section 5.5/Table 5 of the WildfireSpreadTS paper [2]?

[1] Zhang, Yuchen, et al. "Skilful nowcasting of extreme precipitation with NowcastNet." Nature 619.7970 (2023): 526-532. [2] Gerard, Sebastian, Yu Zhao, and Josephine Sullivan. "Wildfirespreadts: A dataset of multi-modal time series for wildfire spread prediction." Advances in Neural Information Processing Systems 36 (2023): 74515-74529.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cEY1 (1/3)

We thank the reviewer for the careful reading and the constructive comments. The main revisions addressing them are:

  1. prediction-step sweeps for one- and two-step training, compared with the step dependence of the stability theorem (W1; Appendix C.2);
  2. real-data ablations with training steps longer than the evaluation step and without the encoder–decoder (W1, W3; Tables 7 and 8);
  3. effective coefficients that combine gg with the linear part of ff (W2; Appendix D);
  4. the settings of every method, persistence on the synthetic tasks, and a standard baseline for each real-data task (Q1–Q3).

Weaknesses

W1. Variable-step prediction is only modestly tested.

More off-grid times. The main tables now also report t3/2t_{3/2}. On KdV, ENNO has the lowest error at t1/2t_{1/2} and t3/2t_{3/2}; on Swift–Hohenberg (SH), it is below OSG-FNO at t1/2t_{1/2}, where KNO and FNO cannot be evaluated (Tables 3 and 6).

Error against step size, one versus two training steps. Appendix C.2 sweeps the prediction step from 0.1τ0.1\tau to 10τ10\tau on KdV and SH for models trained with one and with two step sizes. On KdV, two-step training is more accurate over the whole range, and one of five single-step seeds fails. On SH, the single-step model is more accurate in rollout at multiples of τ\tau but less accurate at other step sizes: training at a single step fits its own grid best, while the second step trades some of this accuracy for accuracy at other step sizes.

Comparison with the theorem. The theorem requires two training steps. It states that if two models are close at the training steps, they stay close at other step sizes, with a bound that is flat between the training steps and grows linearly outside them. The bound contains a constant that is not sharp (Appendix A.1.1 gives an explicit upper bound), so the theorem describes how the difference depends on the step size rather than its magnitude. Figure C.2 estimates this constant empirically on calibration inputs; the held-out differences of the two-step models stay below the resulting curve on KdV and exceed it by at most 18.5% on SH.

Real data. Since the real data have no frames between observations, an ablation trains at two longer step sizes and predicts at the shortest one, the observation interval. On WildfireSpreadTS, these models remain more accurate than KNO and persistence (mean AUC-PR 0.346 against 0.290 and 0.247; Table 8). On SEVIR, they are less accurate than persistence at 5 minutes but more accurate at 25 and 50 minutes (e.g. MSE 20.8×10−320.8\times10^{-3} against 24.3×10−324.3\times10^{-3} at 50 minutes; Table 7).

AuthorsOpenReview ↗

Response to Reviewer cEY1 (2/3)

W2. Interpretability of the model components.

(Submission: coefficient tables (Tables 2 and 5); revision: Tables 2 and 5, Table E.2, and Appendix D.)

Linear and nonlinear components. In ENNO, the linear operator is parametrized by the coefficients of gg, so it can be compared with the PDE coefficient by coefficient. The nonlinear map ff is a neural network on local derivatives: it approximates the nonlinearity but has no term-by-term parametrization, so it is assessed through its output rather than through coefficients. We therefore use the linear coefficients as evidence of the reconstruction ability, not as a guarantee of a term-by-term recovery. That said, a polynomial fit of ff shows that several nonlinear terms are recovered quite accurately: the SH reaction coefficients are 0.975 and −0.991 (true values 1 and −1), and the AD transport coefficients are −0.99 and −1.01 (true value −1) (Tables 5 and E.2).

Highest-order coefficients. Terms above the derivative order of ff can only come from gg. They are recovered closely: the KdV dispersion coefficient c3=−0.9979c_3=-0.9979 against −1, and the SH fourth-order coefficients −0.943 and −1.886 against −1 and −2 (Tables 2 and 5).

Lower-order coefficients. A low-order linear term such as uxu_x can be learned by gg or by ff, because ff also takes the derivatives of the field as input. Our identifiability theorem makes the split unique in principle, but this unique split need not be the way the PDE is usually written. The revision therefore assesses the learned equation differently: we add up the contributions of gg and ff to each linear term and compare these totals with the PDE (the "effective coefficients"; Appendix D). With this new comparison method, we can see that the model recovers the PDE coefficients rather accurately: on AD, the effective first-order coefficients are −0.00005 and 0.0014 (true value 0), and the diffusion coefficients are 0.049 (true value 0.05). Similar results hold on KdV in the centered form of Table 2 (Section 3.1).

W3. Real-data examples and the theory.

(Submission: Theorems 1 and 3; revision: Theorems 1 and 3; Appendix A.1.6, Corollary A.6.)

Theory. We agree that the theorems and the coefficient interpretation apply only to semilinear PDEs. We would argue, however, that such PDEs are widely used in practice, e.g. reaction–diffusion systems, nonlinear waves (KdV, nonlinear Schrödinger), pattern formation (Swift–Hohenberg) and many others (e.g. in PDEBench and APEBench). The theorems do not apply directly to the real data, which use the encoder–decoder and cutoffs that do not satisfy the condition of Theorem 3. While we do not have a rigorous guarantee in this setting, ENNO can still be used in these problems: with the encoder–decoder, ENNO is the most accurate method on both real datasets (Tables 7 and 8). We plan to study the theory in this setting further, and the revision makes an initial effort: Appendix A.1.6 (Corollary A.6) shows that, with a fixed encoder and a Lipschitz decoder, the cross-step stability of the latent model carries over to the decoded predictions.

Operator or encoder–decoder. We ran the ablations requested by the reviewer, with the following results. Without the encoder–decoder, ENNO reaches a WildfireSpreadTS AUC-PR of 0.345, above KNO (0.290) and persistence (0.247); for models trained at two and three days, adding the encoder–decoder raises it from 0.317 to 0.346 (Table 8). On SEVIR, ENNO without the encoder–decoder has MSE 5.2, 10.1 and 13.7 ×10−3\times10^{-3} at 5, 25 and 50 minutes, against 1.3, 5.7 and 9.6 with it; it is more accurate than persistence at 25 and 50 minutes and, at 50 minutes, matches OSG-FNO (13.7) and outperforms pysteps (15.6) (Table 7). These results show that (1) ENNO predicts well on its own, and the encoder–decoder substantially improves it, mostly at short lead times; and (2) our method can predict at step sizes shorter than the training steps with reasonable accuracy (see also W1).

Variable steps. On WildfireSpreadTS, the models trained at two and three days are evaluated at one day, shorter than both training steps, and stay above KNO trained at one day and persistence, with and without the encoder–decoder. On SEVIR, the models trained at 10 and 15 minutes and rolled out in 5-minute steps are more accurate than persistence at 25 and 50 minutes but not at 5 minutes. This is expected: a model that never sees the short step in training cannot, in principle, be as accurate there as one trained on it; the point of this ablation is that it still gives reasonable predictions at a step size it was never trained on. See also our response to W1.

AuthorsOpenReview ↗

Response to Reviewer cEY1 (3/3)

Questions

Q1. Hyperparameters of the benchmarks.

Tuning and defaults. On the synthetic examples, the baseline settings were selected by ablation, with the network scale kept roughly fixed; on the real data, the baselines mostly use default values, and any changes are listed in Table B.4.

Hyperparameter tables. Appendix B.3 lists, for every method, the architecture, number of parameters, training settings and the origin of each setting: Tables B.2 and B.3 for KdV and Swift–Hohenberg, and Table B.4 for SEVIR and WildfireSpreadTS.

TI-DeepONet and NOMAD. We tested both methods extensively, including an additional ablation on advection–diffusion (table below), but could not obtain good results on either synthetic or real data: some variants change the errors quantitatively, but none improves them substantially. Our interpretation is that, unless their scale is increased considerably, both methods, which essentially represent the solution as a neural network of its spatial and/or temporal coordinates, are not well suited to this class of problems.

ModelVariantFrom frame 50From frame 200
NOMADbaseline (5 layers × 128, latent dim. 32)81.6 ± 0.373.8 ± 0.2
3 layers × 15776.2 ± 0.769.9 ± 0.7
latent dim. 6481.9 ± 0.174.1 ± 0.2
40-frame training window81.7 ± 0.173.9 ± 0.0
TI-DeepONetbaseline (4 branch layers, rank 128)30.6 ± 2.323.0 ± 1.8
2 branch layers32.9 ± 6.526.8 ± 5.6
rank 19236.3 ± 7.528.6 ± 7.1
single-step loss weight 10 → 130.3 ± 2.722.3 ± 3.0

Mean relative L2L^2 error (%) over ten prediction steps from the given test frame, mean ± standard deviation over three seeds. All variants have about 2.7×1052.7\times10^5 parameters; each run uses its best-validation checkpoint.

Q2. Persistence baseline.

Thank you for this suggestion. We have added persistence to both synthetic benchmarks in the main text, KdV and Swift–Hohenberg (Tables 3 and 6).

Q3. Standard baselines on the real-data tasks.

On SEVIR, we added pysteps (Pulkkinen et al., 2019), a widely used optical-flow nowcasting method, and on WildfireSpreadTS, logistic regression on 3×33\times3 windows, the reference baseline of the WildfireSpreadTS benchmark (Gerard et al., 2023). Both are commonly used baselines in their fields, and ENNO outperforms both: on SEVIR, its MSE at 50 minutes is 9.6 against 15.6 for pysteps (×10⁻³; Table 7); on WildfireSpreadTS, its mean AUC-PR is 0.376 and 0.427 with one and five input days, against 0.291 and 0.307 for logistic regression (Table 8).

We also tried NowcastNet and similar large nowcasting models, but they could not be run on our computational resources, which also illustrates an advantage of ENNO as a small, lightweight model.

We appreciate that some state-of-the-art models may perform better than ours on these applications, but they address somewhat different problems: ENNO is a neural operator for continuous-valued fields that can be evaluated on different spatial grids, whereas many methods in the WildfireSpreadTS literature predict binary fire masks and operate on a fixed pixel grid. Our comparisons are therefore meant to show that, while retaining these properties of neural operators, ENNO achieves accuracy comparable to standard methods in these fields, not that it is the best method for these applications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the main revisions

We thank the AC and all reviewers for their time and constructive feedback, which we have tried out best to address in the rebuttal. We would be very happy to answer any further questions.

To help the AC and reviewers with their subsequent discussion, we list the main changes in the revision below.

Theory

  • Outlines of the two main proofs, cross-step stability and identifiability (Appendices A.1.1 and A.2.1).
  • A verification that the nonlinear separation condition holds for all three synthetic examples (Appendix A.1.2).
  • A corollary showing that, with a fixed encoder and a Lipschitz decoder, the cross-step stability of the latent model carries over to the decoded predictions (Appendix A.1.6).

Benchmarks and baselines

  • The two-dimensional Swift–Hohenberg scenario of APEBench as the second synthetic example (Section 3.2), and five one-dimensional APEBench scenarios against its built-in baselines (Appendix F).
  • Additional baselines: persistence on both synthetic tasks, pysteps on SEVIR and logistic regression on WildfireSpreadTS.
  • The settings, parameters, wall-clock time, FLOPs and memory of every method (Appendix B).

Additional experiments

  • Five training seeds for every method on the synthetic problems.
  • Errors at t3/2t_{3/2} and t100t_{100} in the main tables, and error curves over the full test trajectories (Appendix C.3).
  • Training without noise and at different noise levels (Appendix C.1 and Table 6).
  • Prediction-step sweeps from 0.1τ0.1\tau to 10τ10\tau, comparing direct steps with rollout and one- with two-step training (Appendix C.2).
  • Ablations of the training step sizes (Appendix C.1), and real-data ablations with longer training steps and without the encoder–decoder (Tables 7 and 8).

Interpretability

  • Effective coefficients that combine the linear operator gg with the linear part of the nonlinear map ff, for every trained model (Appendix D).

Presentation

  • A method diagram (Figure 1), a discussion of splitting schemes in the related work (Section 2.4), the definition of persistence (Section 2.4), and full SEVIR sequences of ENNO, KNO and OSG-FNO (Appendix C.4).

Thanks again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.