acceptodds
Under review as a conference paper at ICLR 2027

Exact Feedback Need Not Help: Correction Compatibility in Frozen Neural Simulators

Abstract

Exact observations improve the coordinates they replace yet can worsen an autoregressive neural simulator's later forecasts. We study correction compatibility: whether a correction improves a frozen simulator's later forecasts. For exact linear dynamics, harm is possible if and only if the propagated, scored error couples corrected and uncorrected coordinates, so exact partial replacement can hurt even a perfect model (related shock from direct insertion is known in data assimilation), and we show that such harm occurs in released neural simulators. A Rayleigh-Bénard checkpoint that beats persistence by about 40% is harmed by two exact corrections selected as harmful on validation, and the harm replicates, with intervals excluding zero, on a previously unused trajectory from the same test cells; the effect is small (about 1.5% of delayed RMSE) and shown for one checkpoint, an existence result. On Supernova simulations, more exact correction helps on average, and all-channel correction helps in every event; in a post-hoc comparison, a channel-restricted exact correction does worse over one step but better over seven than a fraction-scaled all-channel correction with equal immediate repair. How the corrected state is emitted and fed back is a design choice: development data select opposite interfaces for two acoustic checkpoints, and both hold on 200 untouched trajectories. On a turbulence simulator, with the emission mapping fixed, feeding back a learned, constraint-preserving adjustment improves on discarding it, but the approach harms one other system and adds nothing on another. Feedback should be validated by how it enters recurrence, not by its immediate accuracy.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific machine learning, neural simulators, autoregressive models, neural operators, correction compatibility

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_iHkoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the notion of including real (ground truth; correct) state information back into a model during autoregressive simulation. Studying PDE style problems such as Rayleigh-Benard, acoustics, and several others, it analyzes a very interesting phenomenon where inserting true data into a model during autoregression can actually have deleterious effects. Perhaps helping immediately, it can have a degrading effect longer term as the model rolls out -- or vice versa. The paper provides a theoretical justification (or "existence proof") of this phenomenon even in the most restricted and perhaps counterintuitive case. It then proceeds with an empirical evaluation in several settings.

Critical strengths
  • The paper addresses a relevant problem in the field, with autoregressive deep learning approaches being increasingly used as surrogate models for complex physical phenomena. The notion of data assimilation is also interesting and relevant and deserves study, as its implementation can have counterintuitive impacts.
  • Providing theoretical justification for a counterintuitive phenomenon, and then following it up with empirical evidence, is a strength. I have a bit of confusion about both the analysis and the connection between theory and experiments as will be shown below; however, this is still a strength because the hypothesis behind the paper is indeed not obvious on its face.
  • The paper is generally well organized and well written.
Critical weaknesses
  • As mentioned in the Strengths and below, I find the analysis a bit confusing. In a vacuum, the analysis is written clearly and is easy to follow. However, it is unclear how the error dynamics can be treated in a linear fashion and are completely decoupled from nonlinear simulators. I see this with my own eyes and yet for some reason have trouble believing this actually translates to the real problems that are studied in later sections. Perhaps I am just missing something, or perhaps a quick intuitive explanation of this de-coupling (along with the analysis itself) woudl remedy the issue.
  • The paper is relatively thorough in terms of benchmarks. However, it is unclear how the methods and/or results "scale" (generalize?) to other simulator architectures.
  • It is also unclear if there is a direct (for example monotonic) relationship between how "good" the simulator is and how the corrector dynamics work.
Questions for authors
  1. It is unclear how the theoretical justification (proof of universal linear compatibility) relates to the problem at hand. The paper does relatively well to then state a more relevant question, eventually arriving to equation (6). This is all well and good but I am still getting lost because the whole discourse is about proving that harm can be done even on exact linear dynamics, with the implication that this of course holds of inexact and/or nonlinear dynamics. However, what does this theory have to do with actually correcting the harm? I suppose this is a proof of the existence of (non)harm, and then seeks to find average case harm. But all of this is in the linear case, which does not seem to be relevant for most machine/deep learning problems.
  2. At the end of section 2, I believe there is an attempt at answer to the previous point, re: "misspecified learned simulator" and an empirical repair test. What is a misspecified learner? This begs a larger question regarding the approximations of a deep learner: do the paper's conclusions depend on how well a learner has represented the problem? Put in a more simple way, what is the impact of a model's natural prediction error? I assume that more erronous models might be more harmed by corrections than those that are nearly perfect. But is a well-trained and architected model still "misspecified" in the way the authors lay it out? I believe I am missing (or misunderstanding) something fundamental about this approach. The model/simulator itself only "shows up" in the analysis wherever yy is (e.g. ybasey^{base} or other variants of yy). It is interesting and confusing to me how the analysis could then proceed without "worrying" about FF. The analysis in section 2 uses predictions (outputs) of the model (which of course become inputs to the model later), such that everything can be linear in QQ, or PP, or WW, etc. For some reason this seems like a slight-of-hand, as an of the subsequent predictions are then nonlinear maps from xx, which are previous y′sy's. In other words, the analysis depends entirely on (a) the model's output regime and (b) linear correctors; is this the case? How can this be? How can correctors be linear, and a priori known, if the underlying dynamics and/or learned model are nonlinear -- and therefore the error dynamics are nonlinear? I apologize for the long-winded question but I decided to just include most of my notes to give insight into what a first reading might feel like.
  3. Regular U-Nets and/or ConvNeXt U-Nets are used throughout the paper for evaluation (and correct me if I am wrong on this, perhaps I missed a different architecture somewhere). Do the authors believe the papers' conclusions are invariant to architecture? Why U-Nets, or rather way only U-Nets? Nothing wrong with the choice as these are basically SotA, but it is curious to draw general conclusions from such a fixed architecture.
  4. The experimental evaluations and benchmarks are not 1:1. I believe they cannot be due to the underlying nature of the differential equations that are being modeled. I believe the paper would be improved by having a localized, self-contained description of the benchmarks, why and how they're evaluated, and why they are slightly different. Perhaps this could just be in the Appendix (and perhaps it already is, but it is unclear if it is). Appendix A sort of gets at this, but perhaps a paragraph up front there to conceptually describe the benchmarks and why the metrics and protocols must be different, and therefore the reporting in the paper is/are different. This does not need to be a treatise but could help a reader who might be interested in surrogate modeling but does not know each of these benchmarks.
2 comments

it is unclear how the error dynamics can be treated in a linear fashion

Response. They are not linearized. Equation (3) is an exact identity for two actual future rollouts: with baseline future error e and corrected-minus-baseline response r, the squared-error change is 2⟨e,r⟩ + ‖r‖². The realized e and r contain the nonlinear simulator. Only the insertion itself is affine, at the current step; every later step uses the original nonlinear model. Because e needs future truth, this identity diagnoses harm but cannot select an update in advance; hence we recommend validating each feedback route by delayed held-out error, as in our acoustic and turbulence studies (Sec. 4). The separate projector proposition assumes a perfect linear propagator and shows only that exact partial insertion can hurt; it gives no nonlinear guarantee or remedy.

Example: truth (0, 0), forecast (1, −1), next unobserved value (x+y)/2. Autonomy is exact, but correcting x to 0 gives error −0.5. Adding a nonlinear term 0.1x² preserves the harm: squared errors 0.01 versus 0.25.

What is a misspecified learner?

Response. A learned transition that differs from the true dynamics, even on a true input. Well-trained checkpoints are misspecified in this sense. Section 2 shows harm needs no misspecification: with a perfect transition, replacing a cancelling error suffices. Misspecification only changes the realized e and r in Eq. (3), which already accounts for it.

It is also unclear if there is a direct (for example monotonic) relationship between how "good" the simulator is and how the corrector dynamics work.

Response. We claim no monotonic relation; Eq. (3) depends on the signed alignment of e and r, not only on error size. In the nonlinear example above (next y = (x+y)/2 + 0.1x²), adding +0.5 to the learned map raises its one-step error at the true state from 0 to 0.5, yet the same correction now reduces the future error from 0.6 to 0: a worse model can benefit more. Post-hoc (unadjusted), within one checkpoint, across the 35 Rayleigh test files, files with better early autonomous skill had higher correction utility (Spearman 0.364 [0.025, 0.619] and 0.466 [0.188, 0.648] for the two modes), but this is confounded by file difficulty, so we make no causal claim. Post-submission, one-step versus three-step fine-tuning (three seeds, from the released checkpoint) showed no stable difference. On test, mode-6 harm (−1.52%) shrank to at most 0.13% in magnitude, had an interval below zero only in the most competent descendant (58% vs 40% skill), and reversed to help in two (+0.16%, +0.33%; intervals above zero). Mode-9 harm shrank but persisted (−0.10% to −0.39%, versus −1.29%) in four of six descendants (unadjusted intervals below zero); three of six on validation. Descriptively, the least competent (original) checkpoint was most harmed, as you suspected; among the six descendants, harm did not track competence. Competence and fine-tuning are confounded here.

Do the authors believe the papers' conclusions are invariant to architecture?

Response. No, we do not claim architecture invariance; the Rayleigh harm claim is an existence result for one released checkpoint.

Under prespecified 12-mode validation selection per checkpoint, the original was reselected (mode 9; test −1.29% [−1.80, −0.84]) and harm was confirmed in two descendants (mode 9: −0.39% [−0.54, −0.26]; −0.36% [−0.50, −0.17], one multistep-trained); all three survive Bonferroni over seven checkpoints. Both confirmed descendants share one seed, and mode 9 was known beforehand, so this is a robustness check, not independent discovery. Not confirmatory either way: one descendant not selected as harmful on validation still showed test harm (−0.17% [−0.36, −0.07]); one selected descendant (mode 7) did not replicate.

After submission we screened the other released Rayleigh checkpoints against persistence: the official Classic U-Net (−48.85%, validation), TFNO (−487.80%, validation) and FNO (−360.62%, test) all failed our competence check (not compared with The Well's reported metrics), so their correction signs were not interpreted. Of The Well's four released Rayleigh checkpoints, only the ConvNeXt U-Net beats persistence (test +40.15% [27.06, 52.65]; validation +33.37% [19.60, 46.66]), so it is the only released Rayleigh checkpoint whose correction signs we interpret.

On a different system, a competent released Navier-Stokes FNO (post-submission) was helped on average by all three fixed exact Fourier insertions (+4.46% to +9.36% outside both inserted Fourier pairs; adjusted intervals above zero), though 37, 42 and 6 of 200 trajectories worsened. We then applied the Rayleigh validation-selection protocol to this FNO, prespecified: all 40 single Fourier pairs (|k|∞ ≤ 4) had positive validation point estimates (34 with intervals above zero); the most harmful, +0.04% [−0.24, 0.29], had a validation interval crossing zero; on test, no pair had an interval below zero and 36 of 40 had intervals above zero (unadjusted). So harm did not transfer to this FNO; because the system also differs, this cannot separate architecture from system. A controlled same-system architecture comparison remains open.

The experimental evaluations and benchmarks are not 1:1.

Response. Agreed; they differ because each system admits a different observation pattern. All four come from The Well. Rayleigh-Bénard repeatedly inserts one exact mode and scores the state outside a fixed 12-mode span against the free-running model. Supernova makes one channel-subset insertion and scores full-state error. Acoustic scattering compares two ways of feeding the corrected state back under the same pressure observations. Turbulence with gravitational cooling keeps the reported state fixed and compares feeding a learned adjustment back into the model versus discarding it. Their numerical effects are not pooled. The revised PDF adds, at the start of Sec. 4, a table (Table 2) giving each system's observation, schedule, comparator, score, and question.

Reviewer_K1JzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines the effect of feedback in autoregressive physics forecasting systems. Specifically the authors ask the central question: "if the forecast value at time n is corrected what is the impact on the emulator forecast at time n+1". Their conclusion, is that the correction need not always help and may in fact worsen the emulator prediction during rollout. The authors provide a theoretical explanation for this mechanism by linearising the forecast and showcase that this effect can indeed be observed on several numerical experiments on a range of phenomena from acoustic scattering to supernova datasets.

Critical strengths

I think the authors are addressing an interesting question, that has significant practical relevance to the development of learned emulators. As the authors mention, in classical numerical techniques data assimilation (which is the effective analogue, i.e. the correction of trajectory information based on updated data) but in the context of physics emulators this idea has been much less studied. I believe the theoretical analysis presented is interesting and correctly presents the main mechanism for this phenomenon (which is essentially that errors may cancel during accummulation and thus correctors at the current point in time need not necessarily lead to overall better rollout performance). I also believe that the datasets used for the numerical evaluations are broad and sufficient for the current purpose, however as noted below I have some concerns with the breadth of models evaluated.

Critical weaknesses

In my opinion the main weakness of the manuscript is numerical evaluation. Let me be slightly more specific: the main message of the paper is that "correction may cause damage downstream" (paraphrased) so it is perfectly reasonable that any given experiment contains just a single pretrained method and various ways for correcting state information. However, what limits this contribution is that the evaluation focusses exclusively on U-Net architectures, and does not discuss the impact of various training regimes on this correction effect. I think it is very plausible that models pre-trained e.g. with single step loss terms (as U-Nets might reasonably be) suffer from seeing corrected inputs. On the other hand if a model has been trained using multistep loss terms, or perhaps even some noise-injection it is much less clear that the same issues apply. Similarly, it might be that U-Nets are simply unable to capture the full physics of the underlying system thus leading to this amplification. Thus it might be very interesting for the authors to evaluate:

  1. The impact of different training regimes on "correction-sensitivity"

  2. Whether the effect can be observed also in more sophisticated model architectures such as neural operators.

In addition, a small note on presentation:

  1. The abstract refers to a Rayleigh-Benard checkpoint without providing sufficient context, and throughout datasets are refered to simply by adhoc names before a reference to the full dataset is provided. I think it might be helpful if the authors could mention the source of the dataset early on in the manuscript.
Questions for authors
  1. Is it possible to use (3) to make more direct statements on the relationship between correction and error size that would allow and prevent such adverse effects?
3 comments

…such as neural operators.

Response. Post-submission, The Well's official Rayleigh-Bénard FNO, TFNO and Classic U-Net did not beat persistence (validation: Classic U-Net −48.9%, TFNO −487.8%; test: FNO −360.6%); unreconciled with The Well's reported metrics, this screen is inconclusive.

We also tested a released FNO on the 64-by-64 Navier-Stokes benchmark of Li et al. (viscosity 1e-5). It beats persistence by 83.84% [82.96, 84.74]. Exact insertion of the first x, y, or both Fourier pairs improved delayed RMSE outside both pairs by 4.95%, 4.46% and 9.36% (adjusted intervals positive). A prespecified validation selection over 40 single Fourier pairs found none harmful (test: none below zero; 36 of 40 above, unadjusted); Eq. (3) permits harm only when ⟨e,r⟩ > −‖r‖²/2; no FNO correction showed pooled harm, so we have no operator counterexample and claim existence, not prevalence; system and architecture are confounded.

The impact of different training regimes on "correction-sensitivity"

Response. Post-submission, on the harmful Rayleigh-Bénard ConvNeXt checkpoint, we fine-tuned three paired seeds with one-step or recurrent three-step loss. Neither regime showed a stable advantage (multistep minus one-step: mode 6 −0.071 points, adjusted [−0.211, 0.064]; mode 9 +0.035 [−0.062, 0.175]). Fixed-mode harm shrank; validation-selected harm persisted in two of six fine-tuned models, at under a third of the original size. A prespecified rerun of the paper's 12-mode validation selection on all seven models confirmed harm in three, including a multistep-trained model: the original (mode 9, test −1.29% [−1.80, −0.84]) and both models of one seed (mode 9, −0.39% [−0.54, −0.26]; −0.36% [−0.50, −0.17]; 35-file bootstrap; all three survive Bonferroni over seven). Of the other four, three failed the gate and one was null; the most skilful (58%), whose gate failed, showed non-confirmatory test mode-9 harm, −0.17% [−0.36, −0.07]. Skill also rose (40% to 46 to 58%), so skill and harm are confounded, though confirmed harm occurred at 51 to 53% skill. We claim existence in a competent released checkpoint, not survival under training. Test files were reused.

The abstract refers to a Rayleigh-Benard checkpoint…

Response. Agreed. The revised PDF names The Well in the abstract and in the introduction's first paragraph.

Is it possible to use (3) to make more direct statements…?

Response. Yes. Equation (3) is exact for the actual paired nonlinear rollouts. With e the uncorrected later error and r the change the correction causes in that later forecast, squared error changes by 2⟨e,r⟩ + ‖r‖² ≥ ‖r‖(‖r‖ − 2‖e‖). So a nonzero r harms whenever ⟨e,r⟩ ≥ 0 or ‖r‖ > 2‖e‖, and help requires ⟨e,r⟩ < −‖r‖²/2. Only e needs future truth; r comes from two model rollouts, so with an upper validation quantile q of ‖e‖ (assuming exchangeable validation and test cases), ‖r‖ > 2q flags harm whenever ‖e‖ ≤ q, a guarantee averaged over cases, not for cases with large ‖r‖; it is sufficient, not necessary, and misses small harms like ours. So (3) explains but does not predict harm; our submitted signed-probe ranking failed.

The impact of different training regimes on "correction-sensitivity"

Response (noise injection, post-submission). Short answer: noise-injection fine-tuning slightly raised point estimates of correction utility relative to each model's own uncorrected rollout, but it made both uncorrected and corrected forecasts worse, so it is not a mitigation. It also bears on skill: noise-trained models were less skilful than their clean controls (45 to 54% versus 51 to 58% better than persistence on test) yet slightly less harmed on average; because noise targeted the corrected modes, this cannot separate skill from insertion tolerance.

We fine-tuned with independent Gaussian noise on the latest input frame in both corrected modes (6 and 9; targets and evaluation noise-free), with each mode's noise SD equal to its autonomous-validation innovation RMS (the error actually fed back; one amplitude, not a sweep), and three paired seeds with the same training windows and update budget as the clean controls. On the 35 test files, the mean noise-minus-clean change in mode-6 utility was +0.187 points (17-cluster 95% interval [−0.145, +0.430]); mode 9 was +0.201 (two-mode-adjusted 17-cluster interval [+0.059, +0.304]). Under a prespecified but outcome-informed plan, mode 6 is primary (95%) and mode 9 secondary (adjusted 97.5%), both resampling first-state clusters. Both means are driven by one seed (mode 6: +0.49, others −0.02 and +0.09; mode 9: +0.67, others −0.11 and +0.04), so neither establishes improvement across seeds. Full-state utility also rose (mode 6 +0.62 points [0.39, 0.88]), again driven by that seed (+1.65; others +0.03, +0.17); in mode 9 the driving seed's clean control was the most harmed (−0.39%), so regression toward zero is possible; in mode 6 it was helped. Because noise was injected in the corrected modes themselves, greater insertion tolerance is partly expected by design. Autonomous full-state RMSE increased by 15.13%, 11.38% and 9.84% across seeds, and corrected RMSE increased for both modes in every seed.

The modes, clean controls and test files had been used in earlier analyses, so this is a retrospective sensitivity, not a confirmation; intervals condition on three fixed seeds. It tests joint two-mode fine-tuning, not training from scratch. Methods and statistical replays are in the anonymous evidence addendum.

I thank the reviewers for considering my concerns. Overall, I would like to keep my original recommendation, the responses address my concerns in part, and I am happy to retain my recommendation of weak acceptance.

As a minor suggestion, I would encourage the authors to highlight changes in the revised manuscript in colour, where possible, to make the updates easier to identify.

Reviewer_YP3DOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the effect of introducing exact feedback, or exact states, in autoregressive neural PDE solvers.

Critical strengths

The paper studies how exact feedback might interact with autoregressive neural PDE solvers which is relevant to many real-world problems. The writing is fine.

Critical weaknesses

The Title, Abstract, and Introduction hint that exact feedback, or replacing predicted states with exact states, can hurt the rollout. On Rayleigh–Bénard, correcting all 12 modes has no effect; and on Supernova, correcting all channels helps in all cases. Instead, the experiments only show that partially replacing channels or timesteps can hurt--this is unsurprising.

Weak contributions. The paper lists "framing", which would typically be considered "related work", as a contribution. Moreover, in the farming paragraph, the authors identify standard data assimilation methods (e.g., ensemble Kalman filtering, 3D/4D-var), but does not evaluate them. The "controlled evidence" contribution is a minimal existence result; and the "interface validation" contribution suggests that practitioners should tune the feedback scheme on validation data, which is standard practice.

Failed hypotheses. In most cases, the results point in the opposite of the intended direction. On Rayleigh–Bénard, the proposed probe ranking failed; on Supernova, more corrections helps, not harms; on TCG, the authors admittedly did not run an uncorrected rollout, so the value of correction is unmeasured; and all attempts to predict or prevent harm failed. Although I appreciate the transparency about negative results, the results do not support the paper's claims and are not useful.

Questions for authors

See weaknesses.

4 comments

The Title, Abstract, and Introduction hint that exact feedback, or replacing predicted states with exact states, can hurt the rollout. On Rayleigh–Bénard, correcting all 12 modes has no effect; and on Supernova, correcting all channels helps in all cases. Instead, the experiments only show that partially replacing channels or timesteps can hurt--this is unsurprising.

Response. We agree on both counts. The title and the opening of the abstract overstate the finding. The evidence is that partial exact feedback can harm, while full-state correction did not: correcting Rayleigh-Bénard's complete 12-mode span showed no detectable harm, Supernova all-channel correction helped all 96 events, and a post-submission full-field Rayleigh correction on previously accessed test files improved delayed full-state RMSE on every file (77.53% [73.24, 84.67]). The revised PDF rewrites the abstract, Sec. 2 heading and conclusion around partial correction; the ICLR author guide bars title changes until discussion ends, so we propose "Partial Exact Feedback Can Hurt Frozen Neural Simulators" for the final version.

We also agree that partial-replacement harm is not new in principle. Classical DA reports forecast degradation from perfect added observations even with a perfect model (Morss and Emanuel, 2002), and neural-surrogate studies report direct-insertion failures (Penny et al., 2022; Adrian et al., 2025). Our contribution is narrower: with identical exact observations, the better feedback route differed between two official acoustic checkpoints, and the route chosen on development data was also better on 200 untouched test trajectories. Every arm is scored on its raw forecast before insertion, and the Rayleigh primary score excludes the full 12-mode span, so copied values earn no credit. We agree the "framing" paragraph is related work; the revised PDF moves it there (Sec. 5) and lists two contributions.

The "controlled evidence" contribution is a minimal existence result; and the "interface validation" contribution suggests that practitioners should tune the feedback scheme on validation data, which is standard practice.

Response. We agree that validation tuning is standard and is not an algorithmic contribution, and that the Rayleigh harm is an existence result for one checkpoint (−1.52% and −1.29% delayed complement utility for two validation-selected modes, both intervals below zero, 21 of 35 files harmed for mode 9). The acoustic route flip above is why this is more than "tune it": identical observations favoured opposite routes for two checkpoints. The validation score also matters: on Supernova, immediate repair accuracy misranked later value (post-hoc). So one fixed feedback default can be wrong for a given checkpoint, and the check must score delayed error, including uncorrected coordinates.

On Rayleigh–Bénard, the proposed probe ranking failed; on Supernova, more corrections helps, not harms; on TCG, the authors admittedly did not run an uncorrected rollout, so the value of correction is unmeasured; and all attempts to predict or prevent harm failed.

Response. You are right about the probe ranking and about prediction: we have no validated selector. With noisy observations, among standard assimilation schemes only mode-space localization (added after unlocalized filters failed) beat its no-analysis ensemble, and it also beat an inflation-only control, so that gain is assimilation, not inflation (below).

On turbulence with gravitational cooling (TGC), the submitted comparison lacked an uncorrected arm. After submission, on all six public TGC test files (60 trajectories, reused), direct analysis improved delayed unobserved-coordinate RMSE by 1.30% [0.76, 2.48] and full-state RMSE by 4.10% [2.70, 6.50] versus the free run (4 of 60 trajectories harmed on unobserved coordinates, none on full state; Sec. 4.5).

On Supernova, more correction helps on average; some proper channel subsets harm individual events (post hoc).

the authors identify standard data assimilation methods (e.g., ensemble Kalman filtering, 3D/4D-var), but does not evaluate them.

Response. After submission, on previously accessed Rayleigh-Bénard files, we added a static gain, ensemble square-root Kalman filters (EnSRF) and reduced-order 4DVar. Unlocalized, none robustly beat its own no-analysis baseline on full state (static gain: −3.19%, −1.46%; 28 and 26 of 35 files harmed). A tuned 32-member filter scored −6.32% [−8.42, −4.57] against its no-analysis ensemble (30 of 35 files harmed). We then added standard mode-space localization, damping all cross-mode gain by ρ. On validation, a frozen rule chose ρ = 0.1 (inflation 1.3 and model-error scale 1, both grid edges). On test, where noisy insertion of mode 6 hurt (−1.37% [−2.18, −0.78], 32 of 35 files), the localized filter beat its own no-analysis ensemble by 12.84% [6.69, 16.90] on full state and 15.78% [9.04, 19.82] on unobserved coordinates. Because the filter was worse on 18 of 35 files there, a prespecified, paired check used test trajectories 1 to 4, unused by earlier filter studies (Table 10 of the revised PDF):

Localized EnSRF versusUtility [95% CI]Files harmed (of 35)
No-analysis ensemble, model-error scale 16.60% [4.84, 8.40]1
No-analysis ensemble, model-error scale 45.09% [4.07, 6.03]2
Inflation-only control4.84% [4.13, 5.55]0
Noisy insertion of mode 68.09% [5.16, 11.00]1
Same filter with ρ = 0 (no cross-mode update)3.05% [2.57, 3.53]0

Utility is the percent reduction in delayed (predictions 6 to 12) full-state error; positive favours the filter. The no-analysis ensemble is 32 members perturbed by model error and never shown the observation; the inflation-only control is the same ensemble spread by 1.3 about its mean at each event, with no analysis.

So the noisy-observation gain comes from assimilation, not from inflation. Against the inflation-only control, the gain held on both halves (trajectories 1 and 2: 2.54% [1.77, 3.44], 2 files harmed; 3 and 4: 6.86% [6.33, 7.31], 0 harmed). A ρ = 0 filter, which leaves unobserved ensemble means unchanged at analysis, also beat noisy insertion (5.20% [1.98, 8.44]; 7 of 35 files harmed).

Since our claim concerns exact partial feedback, we also tested exact observations (R = 0, prespecified, same settings). The filter beat exact insertion on unobserved coordinates by 12.11% [5.34, 17.03], but a prespecified control, the same inflated ensemble without analysis, did as well; against it, assimilating the exact mode worsened unobserved-coordinate error (utility −1.81% [−2.43, −1.10]; 25 of 35 files harmed; competent trajectories 0 to 2: −1.43%), though full state improved 1.24% because mode 6 is set exactly. In this one-mode, one-draw test, exact partial analysis did not beat inflation alone. One mode, reused files. All in the revised PDF (Sec. 4.6, App. D); per-file harm counts are in the supplementary methods files.

Reviewer_YP3DOpenReview ↗

Response to Official Comment by Authors

I appreciate the effort taken to respond to my concerns. The authors agree with most, or all, of the concerns.

Overall, I believe the work to be fundamentally flawed, including misleading text, weak contributions, and failed hypothesis. So I will have to maintain my score.

Thank you for reading our responses and for the clear summary. We respect your decision and add three short clarifications for the record.

Text. We agreed the submitted framing overstated the finding, and the revised PDF already fixes it: the abstract, Sec. 2 heading and conclusion now state that partial exact correction can hurt, and that full-state correction showed no detectable harm in our tests. Only the title must wait, because title changes are not allowed until discussion ends.

Contributions. We agreed that "framing" is related work and moved it there. The two remaining contributions are evidence, not tuning advice: harm from exact partial correction in a released checkpoint that beats persistence (the harm recurs on a previously unused trajectory and persists under observation noise), and opposite best feedback routes for two official checkpoints under identical observations, both holding on 200 untouched trajectories.

Hypotheses. Our central claim did not fail. The missing TGC uncorrected comparison is now reported, and standard assimilation is now evaluated: even inside an inflated ensemble filter, assimilating the exact partial observation worsened the unobserved state against the same ensemble without analysis (−1.81% [−2.43, −1.10]; Sec. 4.6, App. D). What did not work is predicting harm in advance, which we state as a limitation.

Reviewer_DvV5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this manuscript, the authors study whether a correction of predictions via direct observations can improve or hurt a neural surrogate in autoregressive rollout over a forecast horizon. The experiments cover Rayleigh–Bénard, Supernova simulations, acoustic scattering, and turbulence with gravitational cooling. The main claims are that exact partial corrections can increase forecast error at later lead-times and that short lead-time improvement does not necessarily yield long horizon improvement.

Critical strengths

The setting of neural surrogate prediction with data assimilation is timely. The authors are transparent about the limitations of their study.

Critical weaknesses

The paper is poorly written and hard to understand for a reader. The manuscript is full of nonstandard, new terminology that is vaguely defined and hinders the paper’s quality. 


The manuscript presents results that are unclear in scope. It is unclear what application setting the authors are targeting or what recommendations the authors would make to practitioners based on the results of this paper.

The authors don’t propose a general remedy to the negative effects of correcting for observations.

Questions for authors

Could the authors clarify the scope of the manuscript? What is the recommendation that the authors could make to practitioners based on the results of this paper?

As the authors acknowledge, noisy observations are not studied; could they elaborate on the significance of the results, given that perfect observations are rarely available in practice?

Can the authors compare to more widely-used Kalman-based or variational data assimilation techniques?

2 comments

The paper is poorly written and hard to understand for a reader. The manuscript is full of nonstandard, new terminology that is vaguely defined and hinders the paper’s quality.

Response. We agree that the submission introduces too many new terms too quickly. In data-assimilation (DA) language: An analysis replaces some forecast coordinates (selected modes or channels) with observed values, which we idealize as exact. Emission is which state is reported at that step; carry is which state is fed back as the next model input. Our autonomous rollout is the free run that never receives observations, the usual no-assimilation baseline. The protected complement is the state outside the observed span (for Rayleigh-Bénard, outside a fixed 12-mode span containing every observed mode); we score it separately so that copied observations earn no credit. Correction compatibility asks only whether carrying the analysis lowers later forecast error. Every forecast is scored before that step's analysis. The revised PDF adds this glossary as Table 1, which also defines event, arm, recurrent interface, shadow isolation, and protected carry versus emit-only. "Protected" now names only the scored complement and the protected-carry arm; TGC's 24 observed-span coefficients are now "reserved".

Could the authors clarify the scope of the manuscript? What is the recommendation that the authors could make to practitioners based on the results of this paper?

The authors don’t propose a general remedy to the negative effects of correcting for observations.

Response. Scope: a pretrained autoregressive neural PDE simulator with fixed weights that receives intermittent partial observations during online forecasting.

Recommendations:

  1. Do not accept an analysis because it repairs the observed coordinates; score the following forecasts, especially the unobserved coordinates.
  2. Treat how the analysis re-enters the model as a choice to validate, alongside the autonomous rollout and persistence, at the deployment observation pattern and horizon, then confirm it on held-out trajectories.
  3. If no validated analysis beats its matched no-analysis forecast (deterministic or ensemble), carry that forecast; observations still serve monitoring and checkpoint or interface selection. This is a limitation, not a remedy.

All three recommendations and the scope sentence are now in the revised PDF (Sec. 6).

Direct-insertion shock is known in DA. We add a linear, perfect-model, exact-observation if-and-only-if condition for when partial insertion can raise later error (Sec. 2: replacing a forecast error that was cancelling another one), its measurement in released neural checkpoints, and evidence that the right choice depends on the checkpoint. Under identical acoustic observations and score, development data chose direct feedback for one official checkpoint and a separate autonomous rollout for the unobserved coordinates for the other, and both choices held on 200 untouched test trajectories (Sec. 4.4). A fixed default would have been wrong for one of them.

On remedy: unlocalized EnSRF, 4DVar and the static gain never robustly beat their no-analysis baselines; a mode-space localized EnSRF did, for one mode and noise level (Part 2). We claim no general remedy.

As the authors acknowledge, noisy observations are not studied; could they elaborate on the significance of the results, given that perfect observations are rarely available in practice?

Response. Noise did not remove the harm. Gaussian noise at 0.5 and 1.0 times each mode's validation innovation RMS still worsened delayed complement error versus the autonomous rollout in all four settings (−0.77% to −1.28%; each unadjusted paired-file 95% interval below zero). Full state: −0.91% to −1.27% (negative = harm; Sec. 4.6). Noise attenuated but did not remove the harm (mode 6: −1.52% exact vs −0.77% at 1.0× noise). Noise is synthetic.

Can the authors compare to more widely-used Kalman-based or variational data assimilation techniques?

Response. After submission, on Rayleigh-Bénard with noisy observations of the same modes, we ran three standard schemes with settings selected on validation: a static Kalman/3DVar-style gain, an ensemble square-root filter (EnSRF), and a reduced-rank 4DVar. Unlocalized, none gave a robust full-state benefit over its own no-assimilation baseline; a localized ensemble filter did.

  • Static gain (modes 6 and 9, two noise levels): all four selections passed validation; on test it worsened mode-6 full-state error at both noise levels (−3.19% [−5.56, −1.63]; −1.46% [−2.54, −0.74]; 28 and 26 of 35 files harmed).
  • EnSRF, mode 6: 16- and tuned 32-member filters scored −10.33% and −6.32% [−8.42, −4.57] against their own no-analysis ensembles (28 and 30 of 35 files harmed). After this failure, we localized the gain in mode space, damping every cross-mode update by ρ. A frozen rule on validation files only (36-cell grid) picked a corner: ρ = 0.1, model-error scale 1, inflation 1.3. On test trajectory 0 it beat its scale-1 no-analysis ensemble by 12.84% [6.69, 16.90] but harmed 18 of 35 files, so a prespecified, paired check used trajectories 1 to 4, unused by earlier filter studies (table below; Table 10 of the revised PDF). There, the same filter at scale 4 harmed all 35 files (−23.63%): the benefit requires the small model-error scale.
  • 4DVar: mode 6 gave no benefit; mode 9 passed validation (+1.94%) but not test (+0.54%, adjusted 98.75% [−4.90, 4.27]; 26 of 35 files harmed).

Localized EnSRF on test trajectories 1 to 4 (noisy observation of mode 6):

Localized EnSRF versusUtility [95% CI]Files harmed (of 35)
No-analysis ensemble, model-error scale 16.60% [4.84, 8.40]1
No-analysis ensemble, model-error scale 45.09% [4.07, 6.03]2
Inflation-only control4.84% [4.13, 5.55]0
Noisy insertion of mode 68.09% [5.16, 11.00]1
Same filter with ρ = 0 (no cross-mode update)3.05% [2.57, 3.53]0

Utility is the percent reduction in delayed (predictions 6 to 12) full-state error; positive favours the filter. The no-analysis ensemble is 32 members perturbed by model error and never shown the observation; the inflation-only control is the same ensemble spread by 1.3 about its mean at each event, with no analysis.

So the noisy-observation gain comes from assimilation, not from inflation. Against the inflation-only control, the gain held on both halves (trajectories 1 and 2: 2.54% [1.77, 3.44], 2 files harmed; 3 and 4: 6.86% [6.33, 7.31], 0 harmed). On the unobserved coordinates the filter beat the scale-4 ensemble by 4.73% [3.22, 6.04] and the inflation-only control by 3.69% [2.75, 4.70]. Even the ρ = 0 filter beat noisy insertion (5.20% [1.98, 8.44]; 7 of 35 files harmed): with noisy observations, softly updating the observed mode beats replacing it. With exact observations (prespecified, R = 0), inflating the ensemble without analysis beat exact insertion; adding the exact mode-6 analysis then worsened unobserved-coordinate error (utility −1.81% [−2.43, −1.10]; 25 of 35 files harmed). Inside the ensemble, exact partial insertion still harmed. The revised PDF (Sec. 4.6, Appendix D) reports these alongside the failures; per-file harm counts are in the supplementary methods files.

Comments on the paper

1 comment

We thank all four reviewers for careful, constructive reviews. We were encouraged that reviewers found the question timely and practically relevant to learned emulators (DvV5, YP3D, K1Jz, iHko), the theoretical mechanism interesting and correct (K1Jz), the datasets broad (K1Jz), the paper well organized (iHko), and the reporting transparent (DvV5, YP3D).

What changed

  • Clarity and terminology (DvV5, iHko). A glossary in data-assimilation language (Table 1), a per-system table of observation, schedule, comparator and score (Table 2), and The Well named in the abstract and introduction.
  • Scope and claims (DvV5, YP3D). The abstract, Sec. 2 and conclusion now centre on partial exact feedback; full-state correction showed no detectable harm. "Framing" moved to related work, and Sec. 6 gives the scope and three recommendations for practitioners.
  • Standard data assimilation (DvV5, YP3D). Static gain, ensemble square-root filters and reduced-rank 4DVar, all tuned on validation (Sec. 4.6, App. D). A mode-space localized filter beat its no-analysis ensemble on trajectories unused by earlier filter studies (+5.09% [4.07, 6.03]; one mode, one noise level). With exact observations, adding the exact partial analysis to an inflated ensemble still worsened unobserved error (−1.81% [−2.43, −1.10]), consistent with the paper's mechanism.
  • Noisy observations (DvV5). The Rayleigh-Bénard harm persists under observation noise at both levels tested.
  • Training regimes and architectures (K1Jz, iHko). Harm was confirmed in a multistep-trained descendant of the harmed checkpoint; a competent released FNO on Navier-Stokes was helped by all three exact Fourier insertions we tested, which we report as such.
  • Theory (K1Jz, iHko). As Sec. 2 states, Equation (3) is an exact identity for the actual nonlinear rollouts, not a linearization; our replies add a short nonlinear worked example and the harm bound it implies.

Methods, data provenance and statistical replays for every post-submission result are in the anonymous supplementary material. We are grateful for the discussion, which made the paper clearer and its claims more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.