- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents HyperENS, exploring lightweight ensemble generation via hypernetwork-driven low-rank adapters. HyperENS+ additionally adapts the encoder's query latents. Applied to the Aurora backbone, the approach is highly compute and parameter-efficient, updating just 0.26% of the total weights (~3.24M parameters) within a 16 A100-day training budget. Across 178 initializations from 2024, the resulting 32-member ensemble yields substantial improvements over the frozen parent model, cutting relative ensemble-mean RMSE by 18.8% at 5 days and 24.7% at 10 days. However, this parameter-efficient adaptation does not fully close the gap with dedicated ensemble baselines: fair CRPS remains 9.6% (day 5) and 6.8% (day 10) higher than Aurora 1.5 ENS, and short-lead forecasts remain underdispersed.
Critical strengths
-
A useful practical result under a meaningful training constraint. The paper demonstrates that a frozen global weather model at 0.25° resolution can support effective ensemble forecasting through approximately 3.24 million trainable parameters. The reported gains over the deterministic parent persist to ten days, while the original deterministic forecast remains available by disabling the adapters. This provides useful evidence for extending an existing forecaster with a relatively small incremental training budget.
-
Substantial diagnostic effort and transparent reporting of limitations. The evaluation covers ensemble-mean RMSE, fair CRPS, spread–skill ratios, rank histograms, and spatial spectra. These diagnostics reveal limitations that aggregate skill scores alone would miss. The authors also acknowledge their weaker absolute CRPS than Aurora 1.5 ENS, short-lead underdispersion, checkpoint differences, and several experimental confounds.
Critical weaknesses
- Missing trained baselines leave the method's incremental contribution unclear. Additional task-specific fine-tuning can itself improve forecast accuracy, so gains over the frozen parent do not establish how much improvement is due to the probabilistic method. The paper needs an independently trained deterministic LoRA baseline under an identical experiment setup, with an appropriate deterministic objective and a comparable training budget.
The mean-core model is extracted from an ensemble-trained solution; its parameters were not independently optimised for deterministic accuracy. It is a useful ablation of that solution, but it cannot replace a regular LoRA model trained for deterministic forecasting. The gap between these forecasts therefore does not establish the gain over ordinary deterministic adaptation.
A simpler trained probabilistic adapter is also missing. Given the related methods acknowledged in Section 2, the experiments do not yet show whether the proposed hypernetwork generates better ensemble forecasts than simpler probabilistic adapters. A linear-latent generator or learned Gaussian core should be evaluated under a matched setup. Replacing the learned fluctuations with random noise after training does not establish superiority over a simpler model trained from the outset. Together, these missing comparisons leave unclear both the benefit over deterministic fine-tuning and the benefit of the proposed probabilistic parameterisation.
Questions for authors
-
Deterministic baseline: Please train a regular deterministic LoRA under the matched setup. Then compare its RMSE and MAE with those of the mean-core model and the ensemble mean. This would clarify whether the reported accuracy gains exceed those obtainable through ordinary deterministic fine-tuning.
-
Probabilistic baseline: Please compare with a simpler trained probabilistic adapter, such as a linear-latent generator or learned Gaussian core under a matched setup. This would test the benefit of the proposed nonlinear hypernetwork beyond the current evaluation-time random-core ablation.
-
Encoder attribution: Can HyperENS be continued from the same checkpoint using the same data, training budget, and gradient window as HyperENS+, with only the query-latent adapter disabled? Please specify the gradient-window difference mentioned in the manuscript.
-
Inference overhead: Please report end-to-end latency, member throughput, and peak inference memory for the original backbone and HyperENS/HyperENS+ at matched hardware allocation, precision, resolution, forecast horizon, and batch size. Please state whether core generation or weight preparation is cached or excluded from timing. Repeated deterministic forecasts serve here as a computational reference.
The authors thank the reviewer t7pb for the very concrete questions. Our comment to all reviewers reports the errors we found in our experiments after the submission, and it changes the answer to Q3 in particular.
Q1: "Please train a regular deterministic LoRA under the matched setup."
We did not train this baseline. We agree with your argument completely: the mean core is taken out of an ensemble-trained solution, so its gap to the ensemble mean does not tell how much an ordinary deterministic fine-tuning would give. Since any new training has to be done with the dropout problem fixed, this baseline will be trained together with the corrected ensemble arms and under the same setup.
Q2: "Please compare with a simpler trained probabilistic adapter, such as a linear-latent generator or learned Gaussian core."
Not trained either, neither of the two. You are right that noise inserted after the training does not test a simpler model trained from the outset.
Q3: encoder attribution, and "please specify the gradient-window difference"
The gradient windows were as follows. HyperENS was trained on two 6-h steps with the loss at the each step (6h and 12 h). HyperENS+ was continued from it following the same recipe and then on four steps with the loss at 6 and 24 h. The sentence of Sec. 3.2 about four steps and the loss at 6 and 24 h describes this last stage only, and for HyperENS it is wrong. The data years and the loss weights differ between the arms as well.
The control you describe (HyperENS continued with the data, budget and window of HyperENS+, adapter disabled) was not run. Having said that, we can answer from the other side: with the adapter set to zero in the released HyperENS+ checkpoint, no score changes by more than 0.04 % (16 members, leads up to five days). So we expect that your control would simply reproduce HyperENS+, but this is an expectation and we have not measured it.
Q4: inference overhead
We have no measurement in the matched setup you ask for, so we cannot give this table. What we can state is the structure. Members are not batched, it is one backbone forward per member and step. The hypernetwork (a three-layer MLP, 0.38 M parameters) is not cached and was never excluded from any timing. The adapters are applied unmerged, although the weight change of a member is fixed along its forecast and could be merged once. The adapters and the hypernetwork add 3.2 M parameters (about 13 MB).
The authors thank the reviewer t7pb once again.