acceptodds
Under review as a conference paper at ICLR 2027

Improving Long-Horizon Forecasts with Non-Markovian Corrections to Pretrained Weather Models

Abstract

Unresolved atmospheric processes can introduce history dependence beyond the short input windows used by many weather forecasting models. We introduce GraphCast–Mamba, which augments a frozen, pretrained \(1^\circ\) GraphCast backbone with a trainable Mamba correction branch combining graph message passing and Mamba layers. The branch retains a temporal memory state across successive six-hour forecast steps, learning corrections informed by temporal context beyond GraphCast's two-frame input. Training is confined to the correction branch, using ERA5 data from 2015–2021. After only 2,500 optimizer updates, including warm-up, the selected model achieves substantial improvements relative to the uncorrected GraphCast backbone. It reduces the GraphCast normalized weighted mean-squared error (MSE; GraphCast loss) by 16.3% when aggregated over the full ten-day forecast and by 23.7% at day 10 (240 h after forecast initialization). At this final forecast lead time, root-mean-square error (RMSE) decreases by 10.2–17.1% across all 11 evaluated variable groups, spanning temperature, wind, pressure, geopotential, specific humidity, and precipitation. Separate memory-reset controls support a contribution from persistent temporal context beyond additional trainable capacity. These results demonstrate broad improvements in ten-day weather forecasts through a short correction-training schedule while retaining the pretrained backbone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Non-Markovian dynamics, temporal memory, neural weather forecasting, long-horizon prediction, pretrained model adaptation, state-space models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v96wOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphCast–Mamba, a temporal-memory correction model for medium-range global weather forecasting. First, the authors propose a Mamba-based correction branch that applies memory-guided updates to a frozen GraphCast backbone. They further provide a method to train the correction model over long autoregressive rollouts, ensuring the memory remains effective at extended lead times. Evaluated on ERA5, the proposed method demonstrates consistent improvements in long-lead forecast skill across all evaluated variable groups. The authors also provide ablation studies on memory-reset policies and training tail length.

Critical strengths
  • Lightweighted but effective model: The method improves an existing pretrained forecaster using only a small correction branch. Even with this small addition and short training, the corrected model substantially reduces long-lead forecast error compared with the baseline.

  • Consistent Improvements across variable-group and days: The gains are broad across all 11 variable groups at every lead time from day 5 to day 10. Also, corrected 1° model attains lower long-lead error than the 0.25° GraphCast on a common 1.5° grid.

  • Controlled Memory Ablations: The paper provides an ablation study which separate the effect of temporal memory. This results directly supports the effectiveness of the temporal memory.

Critical weaknesses
  • Potential Effect of Smoothed prediction: As authors mentioned in the limitations section, the RMSE improvements may partially stem from smooth prediction rather than better prediction, lowering RMSE by blurry outputs. It is recommended to clarify this point, to strengthen the effectiveness of the proposed method.

  • Inconsistent Checkpoints Across Comparisons: The comparison with the 0.25° GraphCast uses a model trained with K=22 which differs from the principal model with K = 20. Authors do not explain why a different setting was selected for the comparison. Including the principal model in this comparison would confirm that the claimed advantage over the 0.25° GraphCast holds for the proposed configuration.

Questions for authors
  • Could the authors report energy spectra of the forecasts at long lead times to show whether the RMSE gains come from smoother forecasts?

  • Why is a K = 22 model used for the 0.25° GraphCast comparison instead of the principal K = 20 model? Also, it is recommended to add result for principle model(K=20) for the comparison.

2 comments

We thank the reviewer for the constructive comments and the positive assessment of our correction approach.

Following your suggestions, we have added spectral diagnostics at days 5 and 10 and a fitted output-filtering control to assess the role of smoothing. To address your concern about checkpoint consistency, we have repeated the comparison with 0.25° GraphCast using the principal K=20 checkpoint M1, with paired confidence intervals. The revised results distinguish improvements in forecast error from preservation of spatial detail.

W1. Potential Effect of Smoothed prediction: As authors mentioned in the limitations section, the RMSE improvements may partially stem from smooth prediction rather than better prediction, lowering RMSE by blurry outputs. It is recommended to clarify this point, to strengthen the effectiveness of the proposed method.

Response:

We thank the reviewer for raising the concern about smoothing. We agree that lower RMSE alone does not establish better representation of atmospheric structure. Our new diagnostics show that M1 reduces forecast MSE but has greater aggregate power mismatch relative to ERA5 than raw GraphCast. We now discuss this accuracy–spectral-fidelity tradeoff explicitly and acknowledge that smoothing may contribute to the gains (Section 4.6, Figure 6, and Appendix C). The spectral evidence and fitted output-filtering control are detailed in Q1 below.

W2. Inconsistent Checkpoints Across Comparisons: The comparison with the 0.25° GraphCast uses a model trained with K=22 which differs from the principal model with K = 20. Authors do not explain why a different setting was selected for the comparison. Including the principal model in this comparison would confirm that the claimed advantage over the 0.25° GraphCast holds for the proposed configuration.

Response:

We thank the reviewer for emphasizing consistent use of the principal checkpoint. We agree that the principal comparisons require a consistent checkpoint. The revised comparison uses the selected K=20 model M1 and retains its common-grid RMSE advantage. See Q2 below for the explanation of the earlier choice, the corrected comparison setup, and the results with paired confidence intervals.

Q1. Could the authors report energy spectra of the forecasts at long lead times to show whether the RMSE gains come from smoother forecasts?

Response:

We have added day-5 and day-10 amplitude and coherence diagnostics to distinguish attenuation from spatial-pattern agreement, together with a fitted output-filtering control.

The filter fits a separate spherical-harmonic gain for each lead, variable/pressure level, and total wavenumber by minimizing coefficient-domain squared error on 32 development initializations from 2022. The gains are frozen before evaluation and applied only to GraphCast's reported outputs, leaving autoregressive feedback unchanged. Raw GraphCast, filtered GraphCast, M1, and the matched reset-trained control R1 are evaluated on the same 122 initializations from 2023 and 40 six-hour leads.

Over the full ten-day rollout, M1 achieves 4.02% lower GraphCast-weighted MSE than filtered GraphCast and lower aggregate power mismatch (67.56% versus 78.40%); raw GraphCast has lower mismatch than either (59.47%). Thus, the tested output filter does not reproduce M1's joint accuracy and spectral-fidelity performance, although smoothing may still contribute to its gains. The mismatch metric averages absolute relative-power errors equally over leads, channels, and wavenumbers; it is not a percentage of atmospheric energy lost. Table 6 reports both errors as ratios to raw GraphCast.

Q2. Why is a K = 22 model used for the 0.25° GraphCast comparison instead of the principal K = 20 model? Also, it is recommended to add result for principle model(K=20) for the comparison.

Response:

The original comparison did not use the principal checkpoint consistently. We previously used different checkpoints because their evaluation results were broadly similar, but agree that a unified experimental setup should have been used across the principal comparisons.

We have corrected this by repeating the comparison with 0.25° GraphCast using the original selected K=20 checkpoint M1, as in the variable-group and spectral analyses. Using matched dates on a common verification grid, M1 retains the reported RMSE advantage. The updated results and paired confidence intervals appear in Section 4.4, Figure 4, and Appendix D, Figure 14 and Table 8.

I appreciate authors for their efforts in the rebuttal. I will maintain my score

Reviewer_JvjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article proposes GraphCast Mamba, a non-Markovian correction framework that enhances frozen pre trained GraphCast models using Mamba branches based on trainable graphs. By retaining time memory in the autoregressive prediction step, this method aims to improve long-term weather forecasting. The ERA5 experiment showed that the total GraphCast loss decreased by 16.34% after ten days, and by 23.68% on the tenth day. The RMSE of the 11 variable groups continued to improve. Perform memory reset ablation to study the contribution of persistent time information.

Critical strengths

S1. Substantial improvements in long-horizon forecasting. The method achieves consistent improvements across multiple atmospheric variables, particularly at longer lead times. Improving an existing pretrained model without retraining the backbone is practically valuable.

S2. Meaningful investigation of temporal memory. The carry/reset and reset/reset controls attempt to distinguish the benefits of persistent memory from additional model capacity. The experiments on training rollout length also provide useful insights into autoregressive forecasting.

Critical weaknesses

W1. The core proposition relies on a comparison between persistent memory and memory reset models. However, Figure 6 combines the results of different experimental runs, while Appendix A.2 shows the differences in loss calculation between the main experiments and memory control. In addition, carry/reset introduces training inference mismatch. Therefore, the 10.77 percentage point advantage reported has not yet provided a fully controllable estimate of the independent contribution to temporal memory.

W2. The preliminary assessment only used 32 predictions from the 2022 development set for initialization, which also provides information for hyperparameter selection. Do not provide confidence intervals, multiple training seeds, and independent test results. Although Appendix C includes annual comparisons, these comparisons use different model configurations (K=22), which limits their direct comparability with the main results.

Questions for authors

Q1. Can the author evaluate the main model on a larger independent test set and report statistical uncertainty? This will help establish improved robustness under different atmospheric conditions.

Q2. How does the proposed method compare to GRU/LSTM or fixed history correction models that match parameters? This comparison will clarify whether these gains come from Mamba specific memory mechanisms or more general temporal conditioning effects.

2 comments
AuthorsOpenReview ↗

Following your suggestions, we have removed the carry/reset diagnostic, evaluated the selected K=20 configuration across the 2023 test year, and retained only the carry/carry (M1) versus reset/reset (R1) comparison. We also added five independent retrainings and reported variability and confidence intervals.

We thank the reviewer for the constructive comments.

W1. Controlled memory comparison

We thank the reviewer for highlighting the need for a controlled memory comparison. The original manuscript used carry/carry versus reset/reset to support the benefit of persistent memory. We removed the carry/reset diagnostic, which tested a trained model's dependence on retained memory rather than model superiority, to avoid confusion from its training-inference mismatch.

The revised M1 (carry/carry) versus R1 (reset/reset) comparison uses consistent training and scoring, addressing the loss-calculation concern. Both it and the independent retrainings support a persistent-memory benefit. Q1 below gives the controls, updated estimates, replication and uncertainty; Q2 clarifies the scope of this conclusion.

W2. Independent testing and checkpoint consistency

We thank the reviewer for emphasizing independent testing and consistent comparisons. The selected configuration is fixed before independent full-year testing. Q1 gives the separate training, selection and test data, five retrainings, and retraining versus forecast-date uncertainty.

Earlier checkpoints yielded broadly similar results, but we agree that principal comparisons require a unified setup. The repeated 0.25° GraphCast comparison uses the original selected K=20 checkpoint M1, as in the variable-group and spectral analyses, and retains its RMSE advantage on matched available dates and a common verification grid (Section 4.4, Figure 4, Appendix D).

Q1. Test evaluation and uncertainty

Training uses ERA5 2015-2021, model selection uses 32 initializations from 2022, and principal testing uses 122 initializations spanning 2023. On this test year, five independent retrainings of the fixed configuration, evaluated with checkpoint averaging, yield mean GraphCast loss reductions of 14.17% over the ten-day rollout and 21.03% at day 10. The corresponding sample standard deviations are 0.43 and 0.75 percentage points (Figure 3, Appendix A.5).

For the checkpoint-averaged memory comparison, M1 and R1 share the architecture, optimization, training data and budget, K=20 training tail, and checkpoint-averaging procedure. Both are scored on identical forecast dates and leads using the exact GraphCast loss and a common backbone baseline (Appendices A.1 and A.3). M1 reduces ten-day rollout loss by 14.42%, compared with 9.16% for R1: a 5.26-percentage-point advantage (paired 95% confidence interval [4.70, 5.86]). The day-10 advantage is 8.02 percentage points [7.20, 9.05]. These updated estimates replace the earlier 10.77-point comparison (Section 4.5, Figure 5).

In the independent replication using raw update-2,000 checkpoints for both policies, all five M1 retrainings outperform the local R1 retraining. Their mean rollout loss reduction is 13.51%, with a sample standard deviation of 0.42 percentage points, compared with 9.21% for R1. Even the weakest M1 retraining exceeds R1 by 3.61 percentage points, so the observed advantage is substantial relative to the measured variability across M1 retrainings. R1 has one local training seed, so this comparison does not estimate variability across reset-trained runs. Figure 5 presents this replication separately from the checkpoint-averaged comparison.

Sample standard deviations describe retraining variability. Paired date-block-bootstrap 95% confidence intervals quantify forecast-date uncertainty for fixed checkpoints, not training-seed or between-year uncertainty (Figures 4, 5, and 14; Appendices A.4 and D).

Q2. Comparisons with other temporal architectures

We thank the reviewer for raising this distinction. Our experiment tests whether retaining temporal state improves correction skill within a fixed architecture and parameter count. M1 and R1 share the same Mamba-based branch and differ in their temporal-state policy during training and forecasting. This comparison supports a benefit from persistent state in that branch, but cannot attribute it to a mechanism unique to Mamba.

We have not performed the parameter-matched GRU/LSTM or fixed-history correction experiments needed to establish whether Mamba has an advantage over other ways of incorporating temporal context. We have no a priori expectation that Mamba offers greater expressive power or forecasting accuracy than suitably designed GRU/LSTM corrections in this setting.

We chose Mamba for its selective state-space formulation and computational design, including hardware-aware parallel scanning and efficient recurrent inference (Gu and Dao, arXiv:2312.00752). In our pipeline, autoregressive feedback makes forecast steps sequential, so we do not exploit Mamba's parallel scan across those steps. Repeated GraphCast computation dominates the cost, and we do not claim a measured end-to-end speed advantage here.

Thanks for the authors' rebuttal, I will keep my initial score.

Reviewer_tLBkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper augments a frozen GraphCast model with a trainable correction branch. The branch combines graph message passing and 4 Mamba layers to retain temporal memory across 6-hour steps. Trained for 2,500 steps on ERA5, it reduces 10-day forecast loss by 16.3% and day-10 loss by 23.68%.

Critical strengths

well organized.

Critical weaknesses
  1. The explanation based on non-Markovian closure lacks direct evidence. Residual correction via multi-step training can mitigate rollout distribution shift and error accumulation, yielding performance gains that increase with forecast lead time. The paper fails to rule out this alternative explanation through baselines such as strictly matched multi-step fine-tuning or memoryless correction. Even if the efficacy of persistent memory is established, it remains necessary to distinguish whether it serves to compensate for model errors or to represent the historical dependencies of unresolved physical processes.

2.Necessary baselines are missing. Comparisons with direct multi-step fine-tuning, parameter-efficient fine-tuning, and memoryless correction networks (constrained by the same computational budget) are essential to validate the core claims. Freezing the backbone does not rule out the possibility that performance gains stem from the training of the adapters themselves.

  1. The assessment of prediction quality is incomplete. A general reduction in RMSE fails to distinguish between improvements in dynamics and increased spatial smoothing. The current results lack analyses of predicted fields, power spectra, gradients, and statistics of extreme events; thus, it remains unclear whether the reduction in error comes at the expense of structural integrity or intensity.

4.The conceptual claims exceed the provided evidence.

Questions for authors

Given identical data, training budgets, and multi-step training protocols, what is the independent gain provided by persistent memory compared to direct fine-tuning or residual correction without memory?

1 comment

We thank the reviewer for raising important questions about our baseline comparisons and the role of smoothing. In response, we added a smoothing control and compared GraphCast–Mamba with direct multi-step fine-tuning of GraphCast. Figure 5 retains our earlier comparison between the correction with persistent memory (M1, carry/carry) and its separately trained memoryless control (R1, reset/reset).

W1. “The explanation based on non-Markovian closure lacks direct evidence.”

We agree that a correction trained on multi-step forecasts can learn to compensate for errors that accumulate during rollout. The previous version already compared a correction that carried memory M1, with a separately trained control R1, that reset its state at every step during training and forecasting. Both improve on GraphCast. The M1/R1 comparison asks how much retaining state adds when both models receive the same multi-step training. We give that estimate in Q1 below.

W2. “Comparisons with direct multi-step fine-tuning, parameter-efficient fine-tuning, and memoryless correction networks (constrained by the same computational budget) are essential to validate the core claims.”

To answer your question about multi-step fine-tuning, we note that the published GraphCast training procedure already includes autoregressive fine-tuning. After single-step training, Lam et al. extended the rollout from two to twelve six-hour steps. Our baseline is the released 1° GraphCast_small checkpoint trained on ERA5 from 1979 to 2015. We added this context to Section 3.1. Lam et al., Science (2023), Supplement §4.5

Appendix G, Figure 16 now tests how much further fine-tuning all GraphCast weights achieves on our training data. Section 4.5, Figure 5 gives the memoryless control. An equal-compute comparison with other parameter-efficient methods such as LoRA remains open.

W3. “A general reduction in RMSE fails to distinguish between improvements in dynamics and increased spatial smoothing.”

We fitted a scale-dependent filter to GraphCast outputs on the 2022 development set and evaluated it on 122 forecast starts in 2023. M1 had 4.02% lower ten-day rollout loss than filtered GraphCast. Its aggregate power mismatch against ERA5 was 67.56%, compared with 78.40% for filtered GraphCast and 59.47% for unfiltered GraphCast. At day 5, M1 and R1 had similar amplitude and coherence spectra. At day 10, M1 had lower error but attenuated a broader range of spatial scales (Section 4.6, Figure 6, Appendix C).

The tradeoff also appeared in event detection. At days 5 and 10, M1 had a lower critical success index than unfiltered GraphCast for heat, wind, and 24-hour precipitation above the local 98th percentile and for cold below the 2nd percentile. Spatial gradients remain untested.

W4. “The conceptual claims exceed the provided evidence.”

We use “non-Markovian” because the correction can use forecast history beyond GraphCast’s two input frames. In the matched M1/R1 comparison, retaining that history reduced forecast error. The comparison does not reveal which features of the earlier forecasts M1 used. We do not claim that the learned state captures a physical non-Markovian process in the atmosphere.

Q1. “Given identical data, training budgets, and multi-step training protocols, what is the independent gain provided by persistent memory compared to direct fine-tuning or residual correction without memory?”

Against the memoryless correction, retaining state added 5.26 percentage points of ten-day rollout loss reduction (paired 95% confidence interval 4.70–5.86). On 122 matched 2023 starts, M1 reduced exact GraphCast loss by 14.42% and R1 by 9.16% relative to the same backbone. The day-10 advantage was 8.02 points (7.20–9.05). Both models used the same architecture, parameter count, data, K=20 training tail, optimizer schedule, 2,500 updates, and checkpoint averaging. Both fed back their own forecasts. The intervals describe uncertainty across forecast dates for these fixed checkpoints.

Against direct fine-tuning, a fresh GraphCast–Mamba run reduced pooled ten-day loss by 12.49%, versus 2.19% for full GraphCast, on 16 matched 2023 starts. Both used 2015–2021 ERA5 data, K=20 (five days), 2,500 updates, and final weights without checkpoint averaging. Full GraphCast used a constant learning rate of 3 × 10⁻⁷. Its gain peaked at 4.96% at 5.5 forecast days and fell to 0.99% by day 10, when GraphCast–Mamba achieved 20.10%. The 10.30-point rollout gap compares the two methods as trained. They differ in architecture and learning-rate schedule, and matching update counts does not establish equal wall-clock cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.