acceptodds
Under review as a conference paper at ICLR 2027

Memory Needs Control: Adaptive Memory Gate for Solving PDEs

Abstract

Neural operators have emerged as a powerful data-driven approach for solving time-dependent partial differential equations (PDEs). Among recent advances, memory-augmented neural operators explicitly incorporate past states and have achieved remarkable performance under low-resolution observation settings. However, existing approaches use a fixed memory weight, which sets how much of the past is mixed into the current state. We show that the benefit of memory depends on the resolution and the physical parameters. Empirically, on Kuramoto–Sivashinsky (KS), no single memory weight is optimal across resolutions and viscosities. Theoretically, the Bayes gain of history is bounded by the unresolved spectral energy. We propose AMGFNO, which controls the memory weight with an adaptive memory gate (AMG). The AMG is the product of a content gate and a frequency-aware gate. The gate is learned during training: It opens where much of the spectrum is unresolved and nearly closes where almost all of the spectrum is resolved. On 20 settings spanning 1D, 2D, and 3D PDEs (KS, Burgers', Navier–Stokes, and compressible CFD), AMGFNO is best or tied in 17. On KS at , , AMGFNO reduces nRMSE by 30% (0.036 to 0.025) over the best fixed-memory baseline. The gate is not specific to S4. Across S4, S5, LRU, LSTM, and a causal Transformer memory, the gate lowers the error of the model without the gate in 38 of 45 settings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Memory-Augmented Models, Adaptive Memory Gating, Partial Differential Equations, Time-Dependent PDEs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_zMRQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed an extension of the MemNO framework from Buitrago Ruiz et al., 2025, namely Adaptive Memory Gate FNO (AMGFNO). The novelty of this paper is the adaptive gating of the memory, which includes two gates: one that is content-driven and one that is driven by a precomputed frequency statistic. Additionally, the paper provides a theoretical analysis from a squared-loss Bayes perspective, showing how unresolved spectral energy relates to the potential benefit of using history for an ideal predictor. In 17 out of the 20 settings, AMGFNO is the best or tied. Five baseline models are included, along with the MI-FFNO variant. However, most of these baselines are pre-2024 and a bit outdated for now, given that there are many stronger models available.

Critical strengths
  1. The problem is a well-formulated extension of the Buitrago Ruiz et al., 2025 paper from “whether we need memory” to “how much memory we need.” Based on this paper and previous works, this can be affected by several factors such as the resolution, signal-to-noise level, and the underlying problem itself. Hence, I found the adaptive perspective important.

  2. All experiments seem well-conducted, with proper ablations, multi-seed results, comparisons across different memory layers, and analysis of the additional computational cost.

Critical weaknesses
  1. Although several baselines are provided, most of them are pre-2024. However, given the specific claim of the paper and the inclusion of S4FFNO as the direct baseline, I find this understandable.

  2. The novelty of the method itself is limited. I found the formulation to be heavily incremental over MemNO, and the content-dependent gating strongly resembles GRU-style gating. The frequency-aware part is more specific to this paper, but the overall architectural change is still relatively small.

Questions for authors

None. I am very familiar with this research area, and the authors have already addressed all the potential questions I had in the paper. The only factors preventing me from giving a clear accept are the points mentioned under Critical Weaknesses. However, if the paper turns out to be borderline, I would be willing to argue in favor of it.

1 comment

We thank Reviewer zMRQ for the careful reading and for the willingness to support the paper.


Response to W1: Recent Baseline

We ran DPOT (Hao et al., ICML 2024), a transformer-based neural operator, as a baseline; the results are below. AMGFNO has lower nRMSE in all nine KS settings. DPOT is implemented in 1D and trained from scratch, without pre-training, with the same protocol and a similar parameter count as the other baselines; all models receive only the initial state at test time (Appendix C.4). S4FFNO remains our direct baseline for assessing adaptive memory fusion.

KS, rollout nRMSE (mean ± std over 3 seeds).

ν𝑓DPOTAMGFNO
0.075320.125 ± 0.0080.067 ± 0.006
0.075640.085 ± 0.0090.020 ± 0.002
0.0751280.034 ± 0.0020.006 ± 0.000
0.10320.081 ± 0.0040.025 ± 0.002
0.10640.062 ± 0.0050.006 ± 0.001
0.101280.023 ± 0.0030.003 ± 0.000
0.125320.061 ± 0.0040.016 ± 0.002
0.125640.045 ± 0.0040.004 ± 0.000
0.1251280.019 ± 0.0010.002 ± 0.000

[1] Hao, Zhongkai, et al. “DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training.” International Conference on Machine Learning. PMLR, 2024.


Response to W2: Novelty

Our contribution is the step from “whether memory is needed” to “how much memory is needed”. The frequency-aware gate scales the memory weight by the unresolved spectral energy of the observation condition, the same quantity that appears in the bounds of Eqs. (8)–(9). This factor cannot be replaced by a generic learned gate: with a learned scalar in its place, the gate stays at 0.34–0.39 regardless of the resolution and the error is 11–36% higher (Appendix E.5). Three results support the step to “how much memory". Empirically, the best fixed memory weight varies from 0 to 0.75 across KS resolutions and viscosities. Theoretically, Proposition 1 and Eqs. (8)–(9) relate the Bayes history gain to resolution and unresolved spectral energy. Practically, the gate lowers nRMSE in 38 of the 45 settings across five memory layers, including all 15 at 𝑓=32 (Table 3), with only 0.3–1.1% more parameters (Appendix I).

Reviewer_CNg7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when temporal memory helps neural operators for time-dependent PDEs and proposes an adaptive memory gate. The authors propose AMGFNO, which replaces S4FFNO's fixed residual fusion with an adaptive memory gate that is the product of a content gate (input-dependent) and a frequency-aware gate. Experiments span 1D KS and Burgers, 2D Navier–Stokes, and 2D/3D compressible CFD; AMGFNO is shown to be best or tied in 17 of 20 settings.

Critical strengths
  1. The empirical finding that fixed memory weight is suboptimal is well established and consequential.
  2. The adaptive gate is simple, lightweight, and general across memory layers. The gate adds only about 0.04M parameters.
Critical weaknesses
  1. As stated, the theory motivates the gate but does not predict when the gate will help.
  2. The frequency-aware gate depends on a high-resolution reference that may not be available at deployment.
Questions for authors
  1. Main suggestion (rather then question) is to make some changes in the text. Currently it reads that proposition 1bounds the history gain by the projection residual for an ideal predictor, so no clear bridge to experimental results is evident.
  2. Can you provide some more diagnostic on why at f=128 gate helps Burger and fails at KS? Overall, do you have a feeling why low f are better in terms of memory gain for your method?
  3. The gate adds few parameters, but the paper does not report training time or inference cost relative to S4FFNO. Can you provide those?
2 comments

We thank Reviewer CNg7 for the thoughtful suggestions and questions.


Response to Q1&W1: Bridge between theory and experiments

Thank you for this suggestion. We clarified the theory–experiment connection (Section 2.3, brown). Proposition 1 gives an exact identity for the squared-loss Bayes history gain through the projection residual’s conditional means given history and the current resolved state. Under Assumptions 1–2, Eqs. (8)–(9) provide bounds in terms of resolution and unresolved spectral energy, with PDE-dependent constants. Appendix B.4 (Remark 3) further compares population approximation errors when the history-dependent class contains the current-state class, retaining the latter’s projected-restart approximation error.

These results motivate the empirical condition-level spectral prior. Rollout nRMSE then evaluates the accuracy achieved by the learned fusion mechanism under autoregressive feedback, with memory updated from model-generated inputs.


Response to Q2: Burgers’ vs. KS at 𝑓=128, and why low 𝑓 benefits most

At 𝑓=128, KS (ν=0.125) and Burgers (ν=0.001) have comparable ω^f≈2×10−4\hat{\omega}_f\approx2\times10^{-4}, but mean gate values of <0.001 and 0.002, respectively. A plausible interpretation links this contrast to KS’s fourth-order dissipation, which preferentially damps high-wavenumber modes, and Burgers’ localized steep fronts. Small unresolved components may remain relevant to Burgers front dynamics, allowing history to provide complementary information, whereas accurate current-state prediction leaves less need for memory in KS. Mean gate values summarize fusion weights rather than directly measuring history contributions. Thus, even with comparable ω^f\hat{\omega}_f, the learned gate values and the associated predictive performance may differ across PDEs.

For KS with ν=0.10, ω^f\hat{\omega}_f decreases from 0.58 to 0.083 and 3×10−43\times10^{-4} as 𝑓 increases from 32 to 64 and 128. Coarser observations leave more spectral energy unresolved, increasing history’s potential value. At 𝑓=32, FFNO/S4FFNO/AMGFNO achieve nRMSEs of 0.193/0.036/0.025. Removing the content gate increases nRMSE from 0.025 to 0.037 at 𝑓=32, versus 0.003 to 0.005 at 𝑓=128, supporting input-dependent fusion with a larger absolute benefit at coarse resolution.


Response to Q3: Training and inference cost

Relative to S4FFNO, the gate adds 0.03–0.04M parameters, 1.6–3.2 ms per training iteration, and 0.18–0.78 ms per trajectory at inference.

𝑓ModelParams (M)Train (ms/iter)Inference (ms/traj.)nRMSE ν=0.075ν=0.10ν=0.125
32S4FFNO2.8524.41.830.0860.0360.019
AMGFNO2.8826.02.010.0670.0250.016
64S4FFNO4.9424.11.830.0210.00680.0035
AMGFNO4.9825.92.170.0200.00610.0035
128S4FFNO9.1444.51.820.00720.00340.0031
AMGFNO9.1747.72.600.00610.00290.0021

Timings use one RTX 4090 on KS at batch size 32; nRMSE is the mean over three seeds. At batch 32 both models are launch-bound, so the inference overhead is a fixed cost of extra kernel launches rather than added arithmetic. We added these results to Appendix I: Table 23 reports the numbers above, and Figure 8 plots nRMSE against training and inference cost (brown in the revision).


Response to W2: High-resolution reference at deployment

The high-resolution reference is not needed at deployment; it is used once, at training time. ω^f\hat{\omega}_f is computed from the same high-resolution simulations that are downsampled to form the training set, so it requires no additional data. At inference, the model takes only inputs at the deployment resolution, and no spectral statistic is computed at run time. Low-resolution inputs at deployment are therefore sufficient, as long as the system can be simulated at high resolution for training.

I want to thank th e authors for careful addressing of my (and others') reviewers replies, I will update my recommendation to weak acceptance.

Reviewer_Pr9uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for gating the memory mechanism in neural operators. This is done since the benefits of memory depend on the amount of uncertainty in the system, characterized by factors such as unresolved modes or chaotic dynamics. The authors show that this gating mechanism learns weights that depend on the resolution of the system, and that the gated memory mechanism works well.

Critical strengths
  • The writing is quite good and very clear. The method is well-motivated and it is clear that the amount of gating depends on the system at hand.
  • The proposed adaptive mechanism seems to work well
  • The experiments are well done and are rigorous
Critical weaknesses
  • The positioning of the paper is somewhat odd. There are a few details in the work that make it difficult to see this being used in practice:
    • The method depends on the existence of a set of high-resolution reference trajectories to compute the empirical spectral energy ratio (\omega), which is not usually available in practice.
      • For example, if we want to train a surrogate on some high-resolution simulation data we have, then memory is not generally needed. If we want to train a surrogate on some low-resolution data we have, then we need to generate data at the high/reference resolution to compute \omega. Assuming we can do this (and generating high-resolution data is not prohibitive), then why not train on the higher resolution in the first place?
    • The method performs best at low resolution (at higher resolutions the gate weight goes to zero and is just a regular FFNO). Therefore, the method is really only necessary in cases where spectral modes are unresolved, which doesn’t happen very often in practice. Simulations with unresolved modes are generally unstable so need to be constructed from downsampling higher-resolution simulations; alternatively, I could see this being beneficial in some partial observability setting (say of weather stations or fluid sensors).
  • The gains are somewhat marginal over fixed memory (although this is not a huge concern for me, since the results are quite rigorous)
Questions for authors

I’m looking in the appendix and it seems removing the frequency gate doesn’t harm the performance much. Perhaps there is a way to remove it entirely and improve the content gate a bit more?

2 comments

We sincerely thank Reviewer Pr9u for the thoughtful feedback.


Response to W1: High-resolution reference and training resolution

ω^f\hat{\omega}_f is needed only at training time and requires no additional data. It is computed once from the same high-resolution simulations that are downsampled to form the training set, so inference on low-resolution inputs needs no high-resolution data.

We train at the low resolution because the model must run on the inputs available at deployment, which are low-resolution when they come from coarse or sparse measurements. If high-resolution inputs are available at deployment, one can train at that resolution directly: ω^f\hat{\omega}_f is then small, the gate closes, and AMGFNO is close to FFNO (Table 2, KS at 𝑓=128).


Response to W2: Practical relevance

The method is useful in the partial-observability setting you suggest. With 50% of the sensors missing, the gate opens even at 𝑓=128, where it is at most 0.003 under full observation, and AMGFNO is 5–17% better than S4FFNO in all nine cells (Appendix H, orange).

KS, nRMSE for ν = 0.075 / 0.1 / 0.125, mean over 3 seeds.

𝑓S4FFNOAMGFNOAMGFNO gate
320.208 / 0.183 / 0.1640.194 / 0.174 / 0.1540.81 / 0.83 / 0.84
640.0466 / 0.0359 / 0.03470.0438 / 0.0323 / 0.02890.80 / 0.83 / 0.81
1280.0189 / 0.0121 / 0.01000.0156 / 0.0112 / 0.00950.78 / 0.76 / 0.75

The gate thus follows the amount of unobserved information, not the resolution itself: at 𝑓=128 it closes under full observation and opens when sensors are missing.


Response to Q1: Alternative gate designs

We agree that the content gate carries most of the benefit. However, the frequency-aware gate adds a further gain on top of it. Across the 20 settings of our ablation (Appendix E.1), AMGFNO is lower than or equal to AMGFNO without the frequency-aware gate in 16, and AMGFNO has 11–36% lower error than a variant that replaces it with a learned scalar (Appendix E.5).

Following your suggestion, we tested four richer content gates on KS at 𝑓=32. None gives a clear improvement over the linear gate of AMGFNO: their geometric-mean error is 1–21% higher. We therefore keep the linear content gate. We added these results to Appendix E.6 (Table 14, magenta).

KS at 𝑓=32, nRMSE (mean ± std over 3 seeds).

Content gateν=0.075ν=0.10ν=0.125
AMGFNO (linear)0.0666 ± 0.00560.0254 ± 0.00240.0159 ± 0.0022
Query-only0.0665 ± 0.00470.0255 ± 0.00230.0173 ± 0.0020
Agreement0.0613 ± 0.00380.0256 ± 0.00360.0217 ± 0.0044
Spectral0.0805 ± 0.00460.0316 ± 0.00150.0185 ± 0.0019
Write gate0.0658 ± 0.00990.0264 ± 0.00130.0160 ± 0.0006

Query-only: uses only the current state. Agreement: adds the difference and product of the two inputs. Spectral: one gate per Fourier mode. Write gate: also gates the memory input.

Thank you for the reply. I still maintain that the paper should be accepted.

Reviewer_RTu6OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Neural operators that simulate PDEs on coarse grids miss fine-scale detail, and past states can help recover it. Existing memory-augmented operators such as S4FFNO combine memory with the current state using a weight that is the same for every input. The paper shows that the best weight depends on grid resolution and physical parameters, and supports this with a theoretical bound: the benefit of history is limited by how much spectral energy the grid cannot resolve. The proposed AMGFNO replaces the fixed combination with a gate made of two parts. A content gate computes a memory weight for each sample, position, and channel from the current input. A frequency-aware gate scales this by a value derived from the unresolved energy of the observation condition.

Critical strengths

Studying the impact of memory for neural surrogates is an interesting problem that makes sense in the context of partially observed information.

The proposed contribution is relatively simple and cheap, it could potentially be applied to different surrogates.

Critical weaknesses

Even though I like the idea of ​​paper, I feel proposed setting and experiments do not really show nor convince me about the use of adaptive memory. Taken's theorem clearly state that under partially observation settings, relying on a memory helps to recover the unobserved information, but there should a deeper analysis on different partially observed settings.

The paper presents AMGFNO as a model that adapts memory across observation conditions. But every model is trained on a single resolution and viscosity, so any learned weight already adapts to its setting through training. The experiment that would support the framing, one model trained on mixed conditions, is missing. This could have also been tested on partially observed input with missing sensors in a frame for instance. I think this question is interesting, but deserves a much more detailed analysis, across different settings.

The novelty is therefore a bit limited. The method is a standard sigmoid gate and frequency gate applied to memory fusion. It is

The experiments leave several alternative explanations open. All gated models use an FFNO backbone; testing at least one other backbone (e.g., U-Net or a transformer-based operator with memory and gate) would show whether the method generalizes beyond FFNO-based memory models. The strongest fixed-weight comparison uses convex fusion, so a learnable α that keeps S4FFNO’s residual path is needed to isolate the benefit of per-input gating.

Questions for authors

Have you trained a single model across multiple resolutions and/or viscosities?

How does AMGFNO compare to S4FFNO with a learnable scalar or per-channel alpha that keeps the residual path?

Does the gate transfer to other backbones, such as U-Net or a transformer-based operator with a memory layer?

How much of the gain remains if the same gate is applied without the memory branch, i.e., is the benefit specific to controlling memory?

7 comments

We thank Reviewer RTu6 for the suggestions, which improved our paper.


Response to Q1&W2: A single model across conditions; missing sensors

We trained single models across viscosities and across resolutions, and ran the missing-sensor experiment you proposed. In all three, the gate responds to the condition and AMGFNO has lower error than S4FFNO in most settings. We added the single-model results to Appendix G (Tables 18–19, blue) and the missing-sensor results to Appendix H (Table 21, orange).

One model trained on three KS viscosities is better than S4FFNO in all nine cells (Table 18). nRMSE for ν = 0.075 / 0.1 / 0.125, mean over 3 seeds.

𝑓S4FFNOAMGFNOAMGFNO gate
320.245 / 0.141 / 0.2630.150 / 0.106 / 0.1130.80 / 0.80 / 0.80
640.225 / 0.126 / 0.2710.024 / 0.013 / 0.0100.68 / 0.63 / 0.54
1280.243 / 0.170 / 0.3050.017 / 0.026 / 0.0330.010 / 0.001 / 0.0001

One model trained on three resolutions closes its gate as the resolution increases and has 19–49% lower error at 𝑓=64, 128, and the unseen 96 (Table 19). KS, ν = 0.1, fixed kmax⁡k_{\max}, mean over 3 seeds.

Model𝑓=32𝑓=64𝑓=128𝑓=96 (unseen)
S4FFNO0.0300.0150.0160.037
AMGFNO0.0330.0120.0090.019
AMGFNO gate0.610.400.00020.02

With 50% of the sensors missing, the gate opens even at 𝑓=128, where it is at most 0.003 under full observation, and AMGFNO is 5–17% better than S4FFNO in all nine cells (Table 21). nRMSE for ν = 0.075 / 0.1 / 0.125, mean over 3 seeds.

𝑓S4FFNOAMGFNOAMGFNO gate
320.208 / 0.183 / 0.1640.194 / 0.174 / 0.1540.81 / 0.83 / 0.84
640.0466 / 0.0359 / 0.03470.0438 / 0.0323 / 0.02890.80 / 0.83 / 0.81
1280.0189 / 0.0121 / 0.01000.0156 / 0.0112 / 0.00950.78 / 0.76 / 0.75

Response to Q2&W4: S4FFNO with a learnable memory weight

We ran the comparison you suggested. Both variants keep the residual path of S4FFNO, x←x+αzx \leftarrow x + \alpha z, and learn α as one scalar or as one value per channel. AMGFNO is better than both at 𝑓=32 and 𝑓=128 and comparable at 𝑓=64, so the gain of the gate is not explained by making the weight learnable. In these variants, α is the same for every input after training, whereas the gate of AMGFNO is computed from each input at inference. The gap therefore isolates the benefit of per-input gating. We added these results to Appendix F.3 (Table 17, red).

KS, nRMSE (mean over 3 seeds) for 𝑓 = 32 / 64 / 128.

νS4FFNO (α=1 fixed)Learned scalar αLearned per-channel αAMGFNO
0.0750.086 / 0.021 / 0.00720.085 / 0.022 / 0.00700.074 / 0.020 / 0.00700.067 / 0.020 / 0.0061
0.10.036 / 0.0068 / 0.00340.032 / 0.0063 / 0.00340.029 / 0.0064 / 0.00350.025 / 0.0061 / 0.0029
0.1250.019 / 0.0035 / 0.00310.018 / 0.0034 / 0.00340.017 / 0.0034 / 0.00330.016 / 0.0035 / 0.0021

Response to Q3&W4: Transfer to a U-Net backbone

We evaluate AMG with a U-Net backbone while retaining the S4 temporal-memory branch. S4U-Net uses fixed residual memory fusion, whereas AMGU-Net applies AMG to the same branch. AMGU-Net achieves lower mean nRMSE in eight of the nine KS configurations, with reductions of up to 26%. These results suggest that AMG is applicable beyond FFNO-based memory models. All nine configurations are in Appendix E.8 (Table 16, red).

KS, nRMSE (mean ± std over 3 seeds)

ν𝑓S4U-NetAMGU-NetΔ
0.075320.333 ± 0.0110.317 ± 0.007−4.87%
0.075640.139 ± 0.0320.124 ± 0.015−11.03%
0.0751280.074 ± 0.0010.071 ± 0.003−4.48%
0.125320.100 ± 0.0070.100 ± 0.007−0.09%
0.125640.028 ± 0.0010.028 ± 0.001−0.16%
0.1251280.023 ± 0.0010.018 ± 0.005−24.25%

Response to Q4: The gate applied without the memory branch

The gain largely disappears without memory, indicating that the improvement comes from controlling memory fusion rather than the gated-branch structure itself. With the same gated branch but no memory in it, the error changes by −7% to +17% with no consistent direction (5 lower, 1 unchanged, 3 higher). With memory in the branch, AMGFNO reduces the error of the same FFNO by 5–7× at 𝑓=32. We added these results to Appendix E.7 (Table 15, red).

KS, nRMSE (mean over 3 seeds) for 𝑓 = 32 / 64 / 128. Δ: change from adding the gate to the memoryless FFNO.

ν5-layer FFNO+ AMGFNO gate, no memoryΔ
0.0750.333 / 0.055 / 0.00520.330 / 0.054 / 0.0058−1% / −2% / +12%
0.10.174 / 0.0088 / 0.00280.170 / 0.0082 / 0.0028−2% / −7% / 0%
0.1250.110 / 0.0029 / 0.00190.106 / 0.0034 / 0.0022−4% / +17% / +16%

The two gated models have the same architecture and differ only in whether the gated branch holds memory. The architecture alone therefore gives no gain.


Response to W1: Why memory must be adaptive

The amount of memory needed varies across partially observed settings, so the memory contribution must be adaptive. As you note, Takens’ theorem shows that memory helps under partial observation. Our question is how much memory is needed. The best fixed memory weight ranges from 0 to 0.75 across resolution and viscosity (Fig. 1), and at 𝑓=128 on KS fixed memory even hurts (Table 2).

Following your suggestion, we extended the analysis to further partially observed settings. The results above support the need for adaptive memory in three ways:

  • The gate follows the observation condition. It changes across viscosities and resolutions within a single model, and it opens at 𝑓=128 when 50% of the sensors are missing (Q1&W2).
  • A learned but input-independent weight is not enough. S4FFNO with a learnable α does not match AMGFNO (Q2&W4).
  • The gain comes from controlling memory. The same gate without the memory branch gives no gain (Q4).

Response to W3: Novelty

We agree that the gate itself uses a standard sigmoid function scaled by a frequency-aware term. Our contribution is to show that memory in neural operators must be controlled, to give a measurable signal for it, and to verify that controlling memory produces the gain.

The need is that the best fixed memory weight ranges from 0 to 0.75 across resolution and viscosity (Fig. 1), so any fixed weight loses at one end. The signal is ω^f\hat\omega_f, the spectral energy the observation cannot carry (Appendix C.5): replacing it with a learned scalar leaves the gate at 0.34–0.39 regardless of the resolution (Appendix E.5). The gate also opens when sensors are missing (Appendix H).

For verification, the same gate without the memory branch gives no gain (Appendix E.7), a learnable scalar or per-channel α does not reach the gate (Appendix F.3), the gain transfers to S5, LRU, LSTM, and Transformer memories (Table 3) and to a U-Net backbone (Appendix E.8), and one gated model handles mixed viscosities (Appendix G). The gate adds 0.3–1.1% parameters and about 7% per training iteration (Appendix I).

Thanks for doing these extensive experiments. This answers some of my concerns.

Some follow up questions on this: "show that memory in neural operators must be controlled, to give a measurable signal for it, and to verify that controlling memory produces the gain."

I agree that memory needs to be controlled and your set of experiments seem to show that this help neural surrogates to better generalize. But I guess that this is still based on some assumptions made on the number of frames you consider as input right? Did you try to measure the influence of the number of frames you have access to?

Also, the classical setup is usually initial value problems where you only start from one initial condition at inference, did you try to study the impact of adding more frames as you rollout, starting from one initial condition, or you only assume a fixed number of frames available at inference?

Did you try to do some OOD tests? On conditions unseen during training, does the model require more memory?

Reviewer_y1LLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper tackles the emulation of time-dependent PDEs, where the Markovian property is lost at (low) discrete resolutions. This requires more past information to be propagated. Neural operators can then be 'memory-augmented' to better emulate those systems. The authors argue that the weight of past information should depend on the input, as optimal weighting depends on the physical parameters and the input resolution. They provide evidence for this by (1) sweeping over weights on Kuramoto-Sivashinsky, showing different optimal values for each case, and (2) by providing a bound on the Bayes gain of the history based on the unresolved spectral-energy fraction. They introduce AMGFNO, a variation of Fourier neural operators with an "adaptive memory gate" (AMG) which is the product of a content gate, computed from the memory output and the hidden state, and a frequency-aware gate based on unresolved spectral energy, precomputed on high-resolution training data. They evaluate on 20 settings across 1D to 3D PDEs, where AMGFNO is best or comparable in 17, with the clearest gains on KS at the coarsest resolution.

Critical strengths

The method is cheap to add, with a negligible parameter count and time overhead, and can be added to various architectures. It largely removes the choice between a memory and a memory-less FNO.

The paper has an extensive validation procedure on many datasets, with multiple seeds, from 1D to 3D. The ablations cover many components.

Critical weaknesses

My main concern is the central motivation of the paper being ambiguous. The results demonstrate that a gate can be learned separately for each condition, as the models are trained independently for each setting. Therefore, even though the experiments demonstrate input-dependent gating within an individual setting, they do not demonstrate that a single model can adapt the memory usage depending on the input resolution and physical parameters. This meaningfully weakens the central motivation and claim that can be understood from the abstract, making the method's generalizability uncertain.

Another concern I have is the frequency-aware gate. This is precomputed on high-resolution training data, and therefore is constant within a given condition. This is equivalent to a learned scalar multiplier. Considering the ablation results which show that the content gate is responsible for most of the performance, the authors do not really demonstrate the advantage of this process which requires precomputation compared to a simple learned scalar multiplier gate.

Questions for authors

Q1. Can you confirm the models are trained independently for each PDE, physical parameters, and resolution? If yes, how should one interpret this adaptive memory claim? Is it only at training time given fixed resolution and physical parameters? In that case, are we not losing the advantage of "resolution invariance" of FNOs?

Q2. Can you clarify the advantage of the frequency-aware gate compared to a simple scalar multiplier gate? A small experiment that compares the method with a freely learned scalar would help clarifying.

Q3. The rollouts are only evaluated on short early windows. Could the authors provide errors over longer rollouts and clarify if the advantages persist?

I think the paper tackles an interesting problem with an interesting technique, but the claimed adaptive behavior remains unclear. If Q1 and Q2 are properly addressed, I will consider raising my score.

1 comment

We sincerely thank Reviewer y1LL for the thoughtful feedback, which improved our paper.


Response to Q1&W1: Single-model adaptation and resolution invariance

A single model does adapt its memory usage to the input resolution and physical parameters. To address your concern, we trained one model on multiple conditions and report the results in Appendix G (blue). With one set of weights, the gate varies across viscosities (0.68 / 0.63 / 0.54 for ν = 0.075 / 0.1 / 0.125 at 𝑓=64). It also varies across resolutions (0.61 / 0.40 / 0.0002 for 𝑓 = 32 / 64 / 128). The gate is computed from each input at inference (Eq. 13), so the adaptivity is not limited to training time. The main-table models are indeed trained per PDE, physical parameter, and resolution, following our main baseline S4FFNO (Table 2).

One model trained on three KS viscosities is better than S4FFNO in all nine cells (nRMSE for ν = 0.075 / 0.1 / 0.125, mean over 3 seeds) (Table 18):

𝑓S4FFNOAMGFNOAMGFNO gate
320.245 / 0.141 / 0.2630.150 / 0.106 / 0.1130.80 / 0.80 / 0.80
640.225 / 0.126 / 0.2710.024 / 0.013 / 0.0100.68 / 0.63 / 0.54
1280.243 / 0.170 / 0.3050.017 / 0.026 / 0.0330.010 / 0.001 / 0.0001

One model trained on three resolutions closes its gate as the resolution increases and has 19–49% lower error at 𝑓=64, 128, and the unseen 96 (KS, ν = 0.1, fixed kmax⁡k_{\max}, mean over 3 seeds) (Table 19):

Model𝑓=32𝑓=64𝑓=128𝑓=96 (unseen)
S4FFNO0.0300.0150.0160.037
AMGFNO0.0330.0120.0090.019
AMGFNO gate0.610.400.00020.02

Resolution invariance is lost in the per-condition models because kmax⁡=⌊f/2⌋k_{\max}=\lfloor f/2 \rfloor depends on the resolution, as in S4FFNO. With a fixed kmax⁡k_{\max}, one model covers several resolutions, including the unseen 𝑓=96. With one model trained jointly on all nine conditions (three viscosities × three resolutions), AMGFNO is lower than or equal to S4FFNO in eight of nine cells, by up to 12% (Table 20).


Response to Q2&W2: Frequency-aware gate vs. a learned scalar

We ran the experiment you suggested. The learned-scalar variant replaces the frequency-aware gate of AMGFNO with a freely learned scalar. Everything else is kept the same, including per-condition training. AMGFNO has 11–36% lower error than this variant in all six settings. The gate of the variant ends at nearly the same value regardless of the resolution (0.34–0.39), whereas the gate of AMGFNO opens at 𝑓=32 and closes at 𝑓=128. We added this comparison to Appendix E.5 (Table 13, violet).

KS, nRMSE (mean ± std over 3 seeds); mean gate in parentheses.

𝑓νLearned-scalar variantAMGFNO (frequency-aware gate)Δ
320.0750.0857 ± 0.0063 (0.35)0.0666 ± 0.0056 (0.69)−22%
320.10.0305 ± 0.0021 (0.34)0.0254 ± 0.0024 (0.70)−17%
320.1250.0179 ± 0.0008 (0.34)0.0159 ± 0.0022 (0.70)−11%
1280.0750.0073 ± 0.0008 (0.37)0.0061 ± 0.0002 (0.003)−16%
1280.10.0033 ± 0.0002 (0.37)0.0029 ± 0.0001 (0.001)−12%
1280.1250.0033 ± 0.0004 (0.39)0.0021 ± 0.0003 (<0.001)−36%

We agree that the content gate carries most of the benefit. The frequency-aware gate provides an additional benefit beyond the content gate. AMGFNO achieves equal or lower nRMSE than the variant without the frequency-aware factor in 16 of the 20 ablation settings (Appendix E.1).


Response to Q3: Errors over longer rollouts

The advantage over S4FFNO persists beyond the training window. We rolled out the main-table checkpoints to 48 steps on KS (trained on 23) without retraining. Outside the window, AMGFNO has 24–33% lower error at 𝑓=32 and 61–84% lower error at 𝑓=64 and 128. The full results, including ν=0.1 and 3D CFD, are in Appendix D (Table 8 and Figure 5, violet).

Per-step nRMSE averaged inside ("in") and outside ("out") the training window, seed 155.

ν𝑓S4FFNO inAMGFNO inS4FFNO outAMGFNO out
0.075320.0770.0531.230.94
0.075640.0200.0181.640.64
0.0751280.00750.00540.4670.137
0.125320.0150.0130.600.40
0.125640.00290.00280.2850.064
0.1251280.00230.00170.2740.045

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. Below we summarize the main changes. All changes in the revised PDF are color-marked.

  • Single-model adaptation. We train one model across viscosities and across resolutions, and test it on an unseen viscosity and an unseen resolution (Appendices G and J.3). We also add a setting with 50% of the sensors missing (Appendix H).
  • Source of the gain. We add four controls: a learned scalar in place of the frequency-aware gate, S4FFNO with a learnable scalar or per-channel α, the same gate without the memory branch, and four richer content gates (Appendices E.5–E.7 and F.3).
  • Broader evaluation. We add a U-Net backbone (Appendix E.8), a DPOT baseline, rollouts to 48 steps (Appendix D), and 1 to 8 input frames (Appendix J).
  • Cost and clarity. Appendix I reports training and inference cost relative to S4FFNO. Section 2.3 and Appendix B.4 clarify the link between Proposition 1 and the experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.