acceptodds
Under review as a conference paper at ICLR 2027

How much ensemble does a frozen weather model contain?

Abstract

Data-driven weather models now match numerical weather prediction, and to represent uncertainty, the field trains probabilistic models end to end, at the full training cost of the forecaster. We ask instead how much ensemble skill a pretrained deterministic model provides with its weights frozen, measured against its own forecast. HyperENS trains low-rank adapters on the backbone’s attention projections and a hypernetwork that generates their cores from a Gaussian latent: each new latent is a new member, whose adapter stays fixed along its forecast. The member distribution is trained end to end with an almost-fair CRPS on the forecasts and their power spectra, and no spread amplitude is tuned afterwards. Applied to the pretrained 0.25° Aurora, HyperENS+, which also adapts the encoder’s query latents, trains 3,235,544 parameters, 0.26% of the model, within two days on eight A100 GPUs, and leaves the original deterministic forecast unchanged. Scored against ERA5 on 178 initialisations of 2024, averaging per-variable relative scores over eight variables, its 32-member ensemble mean has 18.8% and 24.7% lower RMSE than the frozen forecast at days five and ten, and its fair CRPS is 0.569 and 0.525 times that forecast’s mean absolute error. Aurora 1.5’s ensemble, a whole-model fine-tune, has the lower absolute fair CRPS: ours is 9.6% and 6.8% higher at days five and ten. The mean spread–skill ratio is within six percent of one from day three, but short leads remain under-dispersed (0.861 at day one) and the rank histograms show residual errors. On an earlier checkpoint evaluated on 418 common initialisations, the adapter’s static part alone is only slightly better than the frozen forecast; most of the gain appears only in the ensemble.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ensemble weather forecasting, probabilistic forecasting, uncertainty quantification, hypernetworks, low-rank adaptation, parameter-efficient fine-tuning, frozen foundation models, model-error perturbation, data-driven weather prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t7pbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents HyperENS, exploring lightweight ensemble generation via hypernetwork-driven low-rank adapters. HyperENS+ additionally adapts the encoder's query latents. Applied to the Aurora backbone, the approach is highly compute and parameter-efficient, updating just 0.26% of the total weights (~3.24M parameters) within a 16 A100-day training budget. Across 178 initializations from 2024, the resulting 32-member ensemble yields substantial improvements over the frozen parent model, cutting relative ensemble-mean RMSE by 18.8% at 5 days and 24.7% at 10 days. However, this parameter-efficient adaptation does not fully close the gap with dedicated ensemble baselines: fair CRPS remains 9.6% (day 5) and 6.8% (day 10) higher than Aurora 1.5 ENS, and short-lead forecasts remain underdispersed.

Critical strengths
  1. A useful practical result under a meaningful training constraint. The paper demonstrates that a frozen global weather model at 0.25° resolution can support effective ensemble forecasting through approximately 3.24 million trainable parameters. The reported gains over the deterministic parent persist to ten days, while the original deterministic forecast remains available by disabling the adapters. This provides useful evidence for extending an existing forecaster with a relatively small incremental training budget.

  2. Substantial diagnostic effort and transparent reporting of limitations. The evaluation covers ensemble-mean RMSE, fair CRPS, spread–skill ratios, rank histograms, and spatial spectra. These diagnostics reveal limitations that aggregate skill scores alone would miss. The authors also acknowledge their weaker absolute CRPS than Aurora 1.5 ENS, short-lead underdispersion, checkpoint differences, and several experimental confounds.

Critical weaknesses
  1. Missing trained baselines leave the method's incremental contribution unclear. Additional task-specific fine-tuning can itself improve forecast accuracy, so gains over the frozen parent do not establish how much improvement is due to the probabilistic method. The paper needs an independently trained deterministic LoRA baseline under an identical experiment setup, with an appropriate deterministic objective and a comparable training budget.

The mean-core model is extracted from an ensemble-trained solution; its parameters were not independently optimised for deterministic accuracy. It is a useful ablation of that solution, but it cannot replace a regular LoRA model trained for deterministic forecasting. The gap between these forecasts therefore does not establish the gain over ordinary deterministic adaptation.

A simpler trained probabilistic adapter is also missing. Given the related methods acknowledged in Section 2, the experiments do not yet show whether the proposed hypernetwork generates better ensemble forecasts than simpler probabilistic adapters. A linear-latent generator or learned Gaussian core should be evaluated under a matched setup. Replacing the learned fluctuations with random noise after training does not establish superiority over a simpler model trained from the outset. Together, these missing comparisons leave unclear both the benefit over deterministic fine-tuning and the benefit of the proposed probabilistic parameterisation.

Questions for authors
  1. Deterministic baseline: Please train a regular deterministic LoRA under the matched setup. Then compare its RMSE and MAE with those of the mean-core model and the ensemble mean. This would clarify whether the reported accuracy gains exceed those obtainable through ordinary deterministic fine-tuning.

  2. Probabilistic baseline: Please compare with a simpler trained probabilistic adapter, such as a linear-latent generator or learned Gaussian core under a matched setup. This would test the benefit of the proposed nonlinear hypernetwork beyond the current evaluation-time random-core ablation.

  3. Encoder attribution: Can HyperENS be continued from the same checkpoint using the same data, training budget, and gradient window as HyperENS+, with only the query-latent adapter disabled? Please specify the gradient-window difference mentioned in the manuscript.

  4. Inference overhead: Please report end-to-end latency, member throughput, and peak inference memory for the original backbone and HyperENS/HyperENS+ at matched hardware allocation, precision, resolution, forecast horizon, and batch size. Please state whether core generation or weight preparation is cached or excluded from timing. Repeated deterministic forecasts serve here as a computational reference.

1 comment

The authors thank the reviewer t7pb for the very concrete questions. Our comment to all reviewers reports the errors we found in our experiments after the submission, and it changes the answer to Q3 in particular.

Q1: "Please train a regular deterministic LoRA under the matched setup."

We did not train this baseline. We agree with your argument completely: the mean core is taken out of an ensemble-trained solution, so its gap to the ensemble mean does not tell how much an ordinary deterministic fine-tuning would give. Since any new training has to be done with the dropout problem fixed, this baseline will be trained together with the corrected ensemble arms and under the same setup.

Q2: "Please compare with a simpler trained probabilistic adapter, such as a linear-latent generator or learned Gaussian core."

Not trained either, neither of the two. You are right that noise inserted after the training does not test a simpler model trained from the outset.

Q3: encoder attribution, and "please specify the gradient-window difference"

The gradient windows were as follows. HyperENS was trained on two 6-h steps with the loss at the each step (6h and 12 h). HyperENS+ was continued from it following the same recipe and then on four steps with the loss at 6 and 24 h. The sentence of Sec. 3.2 about four steps and the loss at 6 and 24 h describes this last stage only, and for HyperENS it is wrong. The data years and the loss weights differ between the arms as well.

The control you describe (HyperENS continued with the data, budget and window of HyperENS+, adapter disabled) was not run. Having said that, we can answer from the other side: with the adapter set to zero in the released HyperENS+ checkpoint, no score changes by more than 0.04 % (16 members, leads up to five days). So we expect that your control would simply reproduce HyperENS+, but this is an expectation and we have not measured it.

Q4: inference overhead

We have no measurement in the matched setup you ask for, so we cannot give this table. What we can state is the structure. Members are not batched, it is one backbone forward per member and step. The hypernetwork (a three-layer MLP, 0.38 M parameters) is not cached and was never excluded from any timing. The adapters are applied unmerged, although the weight change of a member is fixed along its forecast and could be merged once. The adapters and the hypernetwork add 3.2 M parameters (about 13 MB).

The authors thank the reviewer t7pb once again.

Reviewer_1HiHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main idea of this paper is to keep the pretrained Aurora model at 0.25 degree frozen and train some rank-8 low-rank adapters on its Swin attention layers. A hypernetwork then turns Gaussian noise into the adapter weights of each ensemble member, and everything is trained with CRPS. On 2024, the 32-member ensemble mean beats the frozen model by about 19–25% RMSE at days 5 and 10. Spread-skill is close to 1 from day 3, but fair CRPS is still 7–10% worse than Aurora 1.5's full ensemble.

Critical strengths

Strengths:

  • The main benefit of this work is practical operability: the base model stays bit-identical, only 0.26% of parameters are trained, and training takes about 16 A100-days versus about 456 for a full ensemble fine-tune.
  • Clean decomposition between static and stochastic. Strong ablations e.g. spread ladder, random-core control with fair CRPS about 25% worse, showing the gain comes from the learned distribution.
  • The experiments and comparisons seem fair to me, e.g. fair CRPS, size-corrected spread-skill, rank histograms, spectra, paired block-bootstrap CIs.
Critical weaknesses

Weaknesses:

  • I cannot find same-backbone baselines: frozen Aurora with perturbed initial conditions appears only as a two-initialisation pilot. Independent LoRA ensembles, BatchEnsemble, MC-dropout are also missing, so the value of the hypernetwork itself isn't shown.
  • The comparison with Aurora 1.5 is confounded: different base models, data and training stages. HyperENS is also worse in absolute CRPS, and there are no WeatherBench2 comparisons against IFS ENS, GenCast or FGN.
  • Overall rather incremental novelty that does not fundamentally make progress on inference efficiency or ease of deployment. The work assumes someone already frontloaded heavy compute training e.g. Aurora and the authors can leverage that model to build something on top of it.
Questions for authors

Questions:

  • How does frozen Aurora with perturbed initial conditions compare at full scale?
  • How does HyperENS compare with M independent LoRA adapters or BatchEnsemble at matched parameters/compute?
1 comment

The authors thank the reviewer 1HiH. Please read our comment to all reviewers first: it reports errors in our experiments that we found after the submission, and the answers below should be read with it in mind.

Q1: "How does frozen Aurora with perturbed initial conditions compare at full scale?"

Beyond the two-initialisation pilot of the paper, we have this comparison on 34 initialisations of 2024 (00 and 12 UTC). There the frozen Aurora with 32 perturbed initial conditions (ECMWF ensemble perturbations added to ERA5, on 8 of the 13 pressure levels) has a fair CRPS 1.19, 1.11, 1.20 and 1.12 times that of HyperENS+ (the earlier checkpoint) at 6 h, 1, 5 and 10 days. It is over-dispersed at 6 h (spread-skill ratio 1.49) and under-dispersed later (0.82 at day five), while HyperENS+ has 0.77 and 1.06 at the same leads. These numbers will replace the pilot in the next version.

Q2: "How does HyperENS compare with M independent LoRA adapters or BatchEnsemble at matched parameters/compute?"

We do not know. None of these was trained, and neither was an MC-dropout ensemble, so the value of the hypernetwork over them is not shown. You are right that this is the missing part. In fact, the dropout problem from our common comment makes the MC-dropout baseline more important than we thought, since dropout noise in the frozen backbone was a source of spread in our own training and we never evaluated it alone.

W2: the comparison with Aurora 1.5, and no WeatherBench2 comparison

We agree that this comparison is confounded, and we knew it when writing. We could not apply the retrofit to Aurora 1.5 itself, because we did not have all the inputs it needs for training. Thus, the only thing left was to compare the gain of each ensemble over its own deterministic model. It gives an intuition and can hardly be called a controlled comparison. As for IFS ENS, GenCast and FGN, they were not scored in our setup at all: our evaluation year is 2024, and we did not run them on it.

W3: incremental novelty, no progress on inference efficiency

It is true that inference does not become cheaper: it is one backbone forward per member and step, as for any ensemble of this backbone. Starting from a model that somebody has already trained is the setting we chose on purpose, and the question is what ensemble can be added to it with a small training budget and without changing the deterministic forecast. Whether the answer is more than incremental depends on the baselines above, and we do not have them.

The authors thank the reviewer 1HiH once again for the review.

Reviewer_cmkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how much ensemble skill can be extracted from a frozen deterministic weather mode. HyperENS attaches shared rank-8 LoRA templates to 96 attention projections and uses a hypernetwork to map a Gaussian latent to each module's matrix. Trained end-to-end with an almost-fair CRPS and spectral loss in under 16 A100-days, the 32-member ensemble reduces day-5 and day-10 RMSE by 18.8% and 24.7% over the frozen parent.

Critical strengths
  1. The evaluation is quite robust. It encompasses absolute probability scoring, spread–skill relationships, rank histograms, spectral analysis, and ensemble size analysis, alongside a paired, time-blocked bootstrap approach. It also explicitly acknowledges that the absolute CRPS performance is weaker than that of Aurora 1.5.

  2. The mechanistic analysis is comprehensive. By comparing static adapters, stochastic members, ensemble means, amplitude scaling, and stochastic cores, the study provides multifaceted evidence for the value of "learning the perturbation distribution."

Critical weaknesses
  1. The novelty of the proposed method relative to existing stochastic adapters remains to be clarified. It is necessary to demonstrate that the proposed "core" parameterization offers advantages over simpler alternatives, such as trainable Gaussian LoRA or an ensemble of independent adapters.

  2. The probabilistic baseline is insufficient. The ensemble of initial conditions—based on a single, frozen backbone—consists of only two baseline. Consequently, it is not yet possible to fully assess the advantages of the proposed method.

  3. HyperENS and HyperENS+ differ not only in their adaptation locations but also in their training data and configurations; thus, improvements in short-range forecast horizons cannot be attributed solely to query-latent adaptation.

  4. An average spread-skill ratio close to 1 does not prove that the joint distribution or the representation of extreme events is reliable.

Questions for authors

How do Gaussian LoRA, ensembles of independent adapters, and ensembles based on input perturbations compare when given the same budget?

How does a deterministically trained LoRA compare to a "mean-core" model?

Keeping the training data and settings constant, what are the performance gains from adding only query-latent adaptation?

Could you include evaluations using extreme threshold probabilities as well as multivariate and spatio-temporal joint scoring, to assess ensemble quality beyond point-wise CRPS?

1 comment

The authors thank the reviewer cmkD. Our comment to all reviewers reports the errors we found in our experiments after the submission. Your third weakness is one of them, and you pointed at it before we had measured it.

W3 / Q3: "Keeping the training data and settings constant, what are the performance gains from adding only query-latent adaptation?"

You are spot on here. We do not have the run you ask for, with the data and settings kept and only the adapter added. What we have is the opposite test: the adapter removed from the released HyperENS+ checkpoint with everything else untouched. Fair CRPS, RMSE and spread-skill ratio change by less than 0.04 % at every lead up to five days (16 members). So in our checkpoint the gain from the query-latent adaptation is zero, and the short-lead improvement belongs to the differences in data and training configuration that you list. Moreover, the decoder half of the "+" design was never implemented.

W1, W2 / Q1: Gaussian LoRA, independent adapters and input perturbations at the same budget

We did not train a Gaussian LoRA or an ensemble of independent adapters. The random-core test of the paper replaces the learned fluctuation by noise after the training, however we agree that it cannot show an advantage over a simpler adapter trained from the start. For the input perturbations, beyond the pilot of the paper we have 34 initialisations of 2024 (00 and 12 UTC), where the frozen Aurora with 32 perturbed initial conditions has a fair CRPS 1.19, 1.11, 1.20 and 1.12 times that of HyperENS+ (earlier checkpoint) at 6 h, 1, 5 and 10 days.

Q2: "How does a deterministically trained LoRA compare to a 'mean-core' model?"

We did not train a LoRA with a deterministic objective. The mean core is a part of the ensemble-trained solution and cannot stand for it.

W4 / Q4: extreme thresholds, multivariate and joint scores

We agree that a mean spread-skill ratio near one says little about the joint distribution. Our evaluation computes an energy score and a variogram score for T2m and Z500, and we did not report them. On the 178 initialisations shared with Aurora 1.5, the energy score of HyperENS+ is 1 %, 8 to 9 % and 5 to 7 % higher than that of Aurora 1.5 at days 1, 5 and 10, which is the ordering of the CRPS. The variogram score of Z500 is 19 %, 13 % and 7 % higher, so the spatial dependence between the members is where we are furthest from the whole-model ensemble (no confidence intervals were computed for these numbers).

For a small subset of dates we also computed the probabilities of exceeding local climatological percentiles of ERA5 (1995 to 2024) at days 3 to 4 and scored them with the fair Brier skill score against climatology. For T2m above the local 95th percentile the score is 0.41 globally, and for mean sea level pressure below the 5th percentile it is 0.55, against 0.23 and 0.41 for the thresholded ensemble mean. For the pressure lows, the one forecast that we could pair with other ensembles (12 UTC, +96 h) gives 0.55 for HyperENS+, 0.56 for IFS ENS, 0.67 for AIFS ENS and 0.74 for WeatherNext 2. All four are verified against ERA5, which favours our model. Having said that, the probabilities are over-confident for the pressure lows and for the cold tail of T2m, and the tail events are forecast too often, by a quarter to two thirds depending on the event and mostly in the tropics. Once again, this is a small subset, and it says nothing about a season. For the 2024 campaigns the tail evidence there is the rank histograms of the paper, where the extreme bins are over-populated at every lead.

The authors thank the reviewer cmkD once again.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Errors in our experiments, found after the submission

The authors thank the three reviewers for the careful reading of the manuscript. Before the individual answers we have to report three problems in our own experiments. We found them after the submission, and the reviews pointed at some of them independently.

  1. The query-latent adapter of HyperENS+ is inert. We set its output matrix to zero in the released HyperENS+ checkpoint and scored it again with the same latents (16 members, leads up to five days), and no score changed by more than 0.04 %. Thus, the difference between HyperENS and HyperENS+ in the paper has to come from the additional training of HyperENS+: other data years, re-pinned loss weights and a different gradient window. The paper does say that the contrast does not isolate the encoder channel, but the last paragraph of Sec. 4 and its title still credit the channel, and this is wrong.

  2. HyperENS+ is incomplete. The idea behind the "+" is to let the members differ also in how the state is composed into the latent and decomposed from it, which requires adapters on both the encoder and the decoder. We added them to the encoder only and simply forgot the decoder. Sec. 3.1 describes what was actually built, so the description is accurate, but it is not the method we had in mind.

  3. Dropout was active in the frozen backbone during training. We switched the stochastic depth off but overlooked the attention dropout of the backbone (p = 0.1), which stayed on because the model was in training mode. Each member had its own dropout masks in training and none at inference, so the loss was fitted on an ensemble with one more source of spread than the ensemble we evaluate. This concerns every Aurora checkpoint in the paper. It may well be a part of the short-lead under-dispersion, which the paper reads as a property of the perturbation class, and we cannot separate the two without retraining.

As far as we know, the scores in the paper are what the released checkpoints give. Having said that, these checkpoints were trained with the errors above, and a repair means retraining both arms and repeating every evaluation of the paper, which is far beyond the discussion period. For the same reason we did not train or run anything new on the present checkpoints: the results would have to be redone anyway. The few numbers in our replies that are not in the paper come from forecasts that already existed. We expect the qualitative picture to hold, but we cannot promise that the numbers will.

In our opinion the paper should not be accepted in its current state, and we do not ask the reviewers to raise their ratings. We will fix the implementation, add the trained baselines that all three reviews ask for, and submit the work again. The individual replies are there to make the record accurate and to say plainly what we have and what we do not have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.