acceptodds
Under review as a conference paper at ICLR 2027

CRPS-LAM: Probabilistic Regional Weather Forecasting with Continuous Ranked Probability Score

Abstract

Limited-Area Models (LAMs) enable weather forecasting over regional domains at higher resolutions than what is computationally feasible for global models. At such high resolutions, machine learning approaches for weather prediction increasingly rely on ensemble methods to produce probabilistic forecasts. However, existing machine learning LAMs are not scalable due to relying on computationally costly diffusion models or inefficient graph neural networks. We tackle this by introducing a new hybrid CNN/GNN architecture, tailored to the LAM weather forecasting problem. Using this architecture, we construct the DET-LAM deterministic model, producing LAM forecasts both more efficiently and accurately than its graph-based competitor. We then tackle the ensemble forecasting problem, by using this architecture as a backbone for the generative model CRPS-LAM. CRPS-LAM is trained using a Continuous Ranked Probability Score (CRPS) objective, enabling efficient training and sampling in a single forward pass. This yields a speedup of compared to diffusion-based baselines. We evaluate our approach on regional domains in northern Europe, demonstrating that CRPS-LAM produces skillful and well-calibrated forecasts across a range of atmospheric variables.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Weather forecasting, Probabilistic Forecasting, Regional Weather Forecasting, Limited Area Weather Forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_JhULOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies high-resolution regional weather forecasting. It proposes a hybrid model that uses a graph network to read coarse boundary data and convolutional networks to forecast the fine regional grid. The authors train this model as DET-LAM for single forecasts and CRPS-LAM for ensemble forecasts. CRPS-LAM uses a CRPS loss and generates each ensemble member with one model pass per forecast step. On the DANRA dataset, both models outperform the tested graph-based baselines for most variables and lead times.

Critical strengths
  1. Broad evaluation. The paper uses MEPS and DANRA, tests both ERA5 and IFS boundary inputs on DANRA, and includes checks against station observations. It compares with recent graph-based and diffusion-based methods where they can be run.
  2. Good skill at a practical cost. With IFS boundaries on DANRA, CRPS-LAM has lower CRPS than Graph-EFM in 164 of 165 variable and lead-time combinations. Table 1 reports about eight minutes for a 20-member, 72-hour CRPS-LAM ensemble on one A100 GPU. These results support the practical value of the method.
Critical weaknesses
  1. The comparisons do not isolate the benefit of the architecture. Table 1 lists 233 million parameters for DET-LAM versus 23.4 million for Graph-FM, and 251 million for CRPS-LAM versus 14.5 million for Graph-EFM. Graph-FM uses an 800 km boundary region without overlap, while the proposed models use an overlapping 400 km region. Similar inference time is a useful practical comparison, but it does not show how much of the accuracy gain comes from the hybrid design. A smaller version of the proposed model and a comparison with matched boundary inputs would help.
  2. The ensemble fields need a more direct check. Could you test the spatial and temporal structure of individual ensemble members on the DANRA test set? For example, please report the CRPS of 3-hour changes along each member’s forecast trajectory, at several lead times. A comparison with Graph-EFM would show whether CRPS-LAM improves ensemble quality beyond pointwise scores.
Questions for authors
  1. Could you compare smaller DET-LAM and CRPS-LAM models with the graph baselines at similar parameter counts? Could you also compare DET-LAM and Graph-FM with the same boundary region? This would help separate the effects of model size, boundary inputs, and architecture.
  2. Could you measure whether ensemble members preserve realistic spatial and temporal links? For example, compare correlations between nearby locations, changes along a forecast trajectory, or a spatial score.
  3. Section 3 says that using 32 dimensions for edge features saves memory without hurting performance. Could you show an ablation with larger or smaller edge dimensions, reporting both forecast scores and peak memory use?
  4. Could you show how CRPS-LAM’s forecast skill change beyond 72 hours (for example, at 5–7 days) using IFS or ERA5 boundary forecasts? I think [Why we did not scale further] is the reason for diffusion-based models, rather than for CRPS-LAM.
  5. Figure 21 should use different colors for DET-LAM and CRPS-LAM, and the unresolved reference near line 400 should be fixed.
1 comment

We thank the reviewer for the thorough review and the detailed suggestions. We are glad that the reviewer appreciated the broad evaluation, covering MEPS and DANRA, both ERA5 and IFS boundaries and station observations, and found that the forecast skill of CRPS-LAM at a practical cost supports the value of the method. Below we address each weakness and question.

Q1/W1: Model size. We do not claim that the hybrid is better than a GNN at equal parameter count, but that it is much more efficient on current GPUs and software, which lets us use ten times the parameters at a lower inference cost: predicting each time step with DET-LAM (233M) is 30% faster than with Graph-FM and uses less memory. To separate the effect of model size from that of the architecture, we have started training parameter-matched variants DET-LAM-S and CRPS-LAM-S.

Boundary region. Our Graph-FM baseline follows [1], where the overlapping boundary configuration introduced unphysical oscillations and the non-overlapping configuration was therefore recommended. We agree that a comparison with matched boundary inputs adds value, and have started training Graph-FM with the same overlapping 400 km boundary region.

Training even the smaller models requires an estimated 1,500–3,000 GPU hours per run, so we cannot promise results within the discussion period, but we will report all runs that finish in time.

[1] Adamov et al. Building machine learning limited area models: Kilometer-scale weather forecasting in realistic settings. arXiv:2504.09340, 2025.

Q2/W2: We have added an evaluation of the spatial structure of the ensemble members in the revised version (Appendix G). We report the CRPS of average- and max-pooled fields (Appendix G.1), and variogram and energy scores on local patches (Appendix G.2). For the links between nearby locations, we report a distance-weighted variogram score over pairs of grid points up to 100 km apart (Appendix G.2).

CRPS-LAM outperforms Graph-EFM in this evaluation. The variogram and energy scores are lower for all 14 evaluated variables, with a single exception at 72 h. The improvement in pooled CRPS grows when the fields are averaged over larger regions.

Q3: We thank the reviewer for this suggestion. We agree that this claim is currently not justified by any experiments. Our choice of 32 edge feature dimensions was motivated by the findings in [1], where the authors investigate the effect of reducing the edge feature dimensionality and report no noticeable degradation in predictive performance. We adopted this design choice to reduce the memory requirements of our models. In our revision we have updated our claim to "This results in a significant reduction in memory cost. Based on results in the literature for similar settings [1], we do not expect this lower edge dimensionality to impact the performance significantly, and believe that the saved memory is better used for other model components." (line 194).

We agree that an ablation study varying the edge feature dimensionality, with both forecast performance and peak memory usage reported, could provide a more comprehensive evaluation. Unfortunately, we are not certain that we will be able to conduct an additional ablation within the discussion period. Training a single model requires approximately 3,500–7,000 GPU hours, depending on the model configuration, and the available computational resources and limited time do not permit the additional training and evaluation runs required for a meaningful comparison. Furthermore, larger edge feature dimensions lead to out-of-memory errors for our largest model configurations on the available GPUs, preventing us from properly evaluating these settings. We do not expect that such an evaluation would lead to a different conclusion than in [1].

[1] Alet et al. Skillful joint probabilistic weather forecasting from marginals. arXiv:2506.10772, 2025.

Q4: The paragraph "Why we did not scale further" refers to the size of the diffusion baseline, i.e. why we did not train a larger denoiser on DANRA, and not to the forecast length of CRPS-LAM.

The 72 h horizon follows from the limited-area setting and is not a computational limitation of CRPS-LAM. At 5–7 days the information in the high-resolution initial state has largely left the domain, so the forecast is mainly determined by the global forcing entering through the boundary, and an evaluation would mostly measure the skill of that forcing. In addition, CRPS-LAM is fine-tuned with rollouts of up to 36 h and calibration degrades beyond the training rollout length (new Appendix F), so a meaningful 5–7 day evaluation would probably also require fine-tuning on longer rollouts, which we cannot complete within the discussion period.

Q5: Thank you for noticing this. We have fixed the unresolved reference and changed the color of DET-LAM in the plots.

Reviewer_NkCAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an interesting idea: use a hybrid backbone for regional weather forecasting. Regional forecasting is generally challenging due to e.g. boundary conditions, limited data availability and local phenomena. That makes it challenging for different deep-learning based forecasts -- so the natural question is if different models (e.g. a CNN and a GNN) can be complementary to a certain extent. The authors propose to use a GNN to encode coarse boundary conditions and a CNN to handle regional interior. The model is both trained as a deterministic model using an MSE-style loss function and a probabilistic one based on CRPS and latent noise. The main benefit of the proposed ensemble is computational speedups compared to Diffusion-LAM (39x faster at similar performance) and fidelity (beats models like Graph-FM on 2.5km resolution).

Critical strengths

Strengths:

  • Realistic setup for regional forecasting where the boundaries come from different grids at coarser time step. Also the ensemble is trained on ERA5 and then tuned on IFS, checked against observations and an operational numerical model
  • The efficiency gain compared to Diffusion-LAM is very noticable
  • Gains over models like Graph-FM are consistent with good reporting of compute/training settings
Critical weaknesses

Weaknesses:

  • The model size looks like a confounder here: 233-251M params vs. baselines with 14.5-23M params is difficult to compare without ablations at equal capacity. Essentially, in ViT-world, this would be a comparison of a tiny/small model against an almost large model.
  • The novelty is limited for an outlet like ICLR. The CRPS training plus noise injection follows existing work and the boundary handling too. So the primary contribution is to marry the two. That in itself is a nice idea, yet I'm not convinced it's enough for one or the premier ML conferences.
  • I cannot really see that the claim that ensemble members are spatially coherent is signficantly backed up. E.g. the metrics are only reported per grid point, but no energy/variogram scores and no evaluation of extremes.
Questions for authors

Questions:

  • I would be interested in the following ablations: How does a non hybrid model score at same size (e.g. only GNN/CNN)? How do the benchmarks look when the model capacity is reduced e.g. less then 30M parameters?
  • Can you add energy or variogram scores and extreme event metrics?
  • Would ensemble IFS boundaries or longer fine-tuning rollouts keep calibration stable beyond 36h?
1 comment

We thank the reviewer for the careful and constructive review, and are glad that the realistic regional forecasting setup and the efficiency gain over Diffusion-LAM were appreciated. Below we address each weakness and question.

W1/Q1: We agree that the parameter counts differ by an order of magnitude, and we should have motivated this choice more clearly in the paper. We do not claim that the GNN-based baseline is a worse model at equal parameter count: a GNN is more flexible, and a model equivalent to DET-LAM could in principle be implemented as one. Our claim is that the hybrid CNN-GNN architecture is far more efficient on current hardware and software, which allows a ten times larger model at a lower inference cost: predicting each time step with DET-LAM (233M) is 30% faster than with Graph-FM (23.4M), and DET-LAM uses less memory at inference time. The comparison in the paper is therefore at matched inference cost rather than at matched parameters, and we see the size difference as a possibility of the design rather than a confounder.

We agree that it is informative to separate the effect of the larger model size from that of the architecture itself. A GNN with 250M parameters would probably improve the graph-based models, but it does not fit in the memory of our GPUs. We have instead started training two parameter-matched variants: DET-LAM-S (matching Graph-FM) and CRPS-LAM-S (matching Graph-EFM). Each run requires an estimated 1,500–3,000 GPU hours, so we cannot promise results within the rebuttal period. We will report them in our next response if the runs finish in time.

W2 We agree that the individual components build on established ideas. However, the novelty lies in the flexibility of our architectural design framework. This explicit separation allows us to tailor each component to its underlying grid structure, combining flexible boundary handling with efficient large-scale processing. We believe this architectural contribution, together with the probabilistic training approach, provides a meaningful advance over existing methods. Beyond the architectural contribution, we train and evaluate Graph-EFM in a more realistic limited-area forecasting setting, extending its evaluation beyond the simplified scenarios considered in previous work. Furthermore, we investigate fine-tuning on IFS data for all models, bringing the framework closer to operational regional forecasting. Together, these contributions provide both methodological and empirical advances over existing work.

W3/Q2: We have added the requested scores in the revised version (Appendix G). The appendix contains (i) the CRPS of average- and max-pooled fields (Appendix G.1), (ii) variogram and energy scores on local patches and a distance-weighted variogram score over pairs of grid points up to 100 km apart (Appendix G.2), and (iii) Brier scores for local extremes of 2 m temperature, 10 m wind speed and mean sea level pressure (Appendix G.3).

CRPS-LAM outperforms Graph-EFM in all three evaluations. The variogram and energy scores are lower for all 14 evaluated variables, with a single exception at 72 h, and the Brier score is lower in 17 of the 18 combinations of event and lead time. The improvement in CRPS grows when the fields are averaged over larger regions.

Q3 Yes, we expect both to help, and we have added results for the rollout length in the revised version (Appendix F).

Longer fine-tuning rollouts. The IFS fine-tuning consists of a stage with 4-step rollouts (12 h) followed by a stage with 12-step rollouts (36 h). In the new Appendix F ("Effect of the Fine-Tuning Rollout Length on Calibration") we compare the spread-skill ratio (SSR) of the models after the two stages. For CRPS-LAM, the longer rollouts clearly extend the calibrated range. Calibration remains stable somewhat beyond the rollout length used in training, and we expect that fine-tuning on even longer rollouts would extend this further, at a higher training cost.

Ensemble IFS boundaries. While we do not have direct experiments on this, we think that it would help. CRPS-LAM currently receives a single boundary forecast, so the uncertainty of the boundary conditions is not represented in the ensemble. Since the forecast depends more on the boundary forcing at longer lead times, we expect this to contribute to the underdispersion at these lead times, and that using a different IFS ensemble member as boundary forcing for each CRPS-LAM member would increase the spread there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.