acceptodds
Under review as a conference paper at ICLR 2027

Hyan: State-Adaptive Hidden Perturbations for Efficient Ensemble Weather Forecasting

Abstract

Reliable weather forecasting requires both accurate predictions and calibrated uncertainty estimates, yet diffusion-based ensemble methods often incur high inference costs due to iterative sampling. We propose Hyan, an efficient ensemble weather forecasting model with state-adaptive noise injection into intermediate representations, and Hyan DE, its deep-ensemble variant. At each injection layer, a lightweight bottleneck network uses local latent features to predict mixing coefficients that control the relative contributions of latent features and stochastic noise. This mechanism produces spatially varying, state-dependent perturbations and requires only one forward pass per ensemble member at each forecast step. On WeatherBench2 at resolution, Hyan DE reduces CRPS by an average of 7.92% relative to IFS ENS over the first five forecast days and outperforms ArchesWeatherGen at only 10.76% of its inference cost. Ablation experiments show that explicit state conditioning improves probabilistic forecast skill and the agreement between ensemble spread and forecast error. Visualizations further reveal distinct spatial modulation patterns across perturbation layers, vertical groups, and forecast lead times.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ensemble weather forecasting, probabilistic forecasting, uncertainty quantification, stochastic perturbations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mzg1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Motivated by the high cost of diffusion-based probabilistic DLWF models, this paper proposes Hyan, an efficient ensemble weather forecasting model with state-adaptive noise injection into intermediate representations, and Hyan DE, its deep-ensemble variant. A lightweight bottleneck network produces spatially varying, state-dependent perturbations, requiring only one forward pass per ensemble member. On WeatherBench2, Hyan DE reduces CRPS by 7.92% relative to IFS ENS and outperforms ArchesWeatherGen at only 10.76% of its estimated inference cost.

Critical strengths
  1. The proposed state-adaptive perturbation mechanism enables efficient ensemble forecasting without iterative diffusion sampling.

  2. This paper is easy to follow.

Critical weaknesses
  1. My biggest concern is the insufficient motivation and physical justification for intermediate perturbations proposed in this work. Typically, ensemble forecasting aims to characterize uncertainty in physically plausible atmospheric evolution, rather than merely introduce variability into predictions. For example, the recent GenCast model introduces Gaussian noise as part of its input because it assumes that Gaussian noise can characterize the uncertainty inherent in weather states and the noise in observed weather data. Take diffusion-based models as another example; they inject noise directly into weather data in a finer way. However, in this work, Hyan injects stochastic perturbations into intermediate latent representations without sufficiently justifying what sources of physical uncertainty these perturbations represent or whether the resulting forecasts remain physically plausible and interpretable. The authors should conduct further investigations into the physical plausibility and interpretation of these perturbations and provide a more fundamental justification for the proposed method before resubmission.

  2. It remains a concern about fair computational comparisons (the key motivation of this work). The reported inference-cost advantage relies on approximate hardware equivalences based on theoretical peak computational capabilities rather than measured end-to-end performance. Several baselines also use different spatial resolutions.

  3. Several design choices, including the bottleneck dimension, perturbation locations, spherical noise spectrum, and autoregressive rollout schedule, lack systematic sensitivity analysis. Their effects on forecast skill and training stability remain unclear.

  4. The main CRPS improvements are reported without confidence intervals or significance tests. In particular, the 1.46% improvement over ArchesWeatherGen requires stronger statistical evidence.

  5. The experiments mainly evaluate aggregate forecasting metrics, leaving the model's ability to represent rare events, such as tropical cyclone prediction. This might be a necessary empirical investigation which has been done in other DLWF models such as GenCast for justifying the effectiveness of the proposed approach.

Questions for authors

Please refer to the weakness. I am open to increasing my scores if the authors can fully address my concerns.

4 comments

Thank you for your comment.

W1. My biggest concern is the insufficient motivation and physical justification...

We respectfully disagree on this point. GenCast and DPMs share the same underlying principle, and we are not aware of evidence that denoising-based sampling leads to more physically plausible forecasts. GenCast, for instance, still shows long-term drift on many metrics.

We would also note that our method belongs to a different line of work, namely efficient ensemble forecasting (FourCastNet3 [1], FGN [2], AIFS-CRPS [3]). Prior methods do not account for the local state and rely on fixed or global perturbations, which limits how well they model the optimal perturbation direction. We therefore propose an efficient perturbation method conditioned on the latent state. Ablations show that it improves accuracy, and our visualisations show that perturbation strength correlates to some degree with humidity.

W2. It remains a concern about fair computational comparisons (the key motivation of this work)....

We thank the reviewer for this correction. For GenCast, ArchesWeatherGen and NeuralGCM, the average per-step, per-member inference time on the A100 is taken from the two papers cited above [4, 5]. For UCast, Hyan and Hyan DE, we instead measured the inference time directly on an A100 in FP32 using torch.compile, with compilation and warm-up completed and the average taken over repeated runs. The total inference time per forecast is summarised below. These new results have been added to Fig. 1, and the description of the conversion has been updated accordingly.

GenCastArchesWGNeuralGCMUCastHyanHyan DE
Infer (A100 s)48.03.50.960.2180.3690.369

W3. Several design choices, including the bottleneck dimension, perturbation locations...

We sincerely appreciate this important suggestion. The core contribution of this paper lies in how to construct a better perturbation mechanism that makes the forecasts more accurate, rather than in the engineering choices of the backbone, and this is why we have only briefly covered these aspects. In fact, most of the architectural design is not original to this paper. The backbone follows ArchesWeatherGen [4], and the noise components follow FourCastNet3 [1], GenCast [6] and others, all of which are already in common use. Our experiments therefore focus on ablating the mechanism of noise injection and on the corresponding analyses, namely the comparison between conventional perturbation methods and the method proposed here (Sec. 6.1) and the visualisation of the perturbation distribution (Sec. 7).


[1] Bonev, Boris, et al. "FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale." arXiv preprint arXiv:2507.12144 (2025).

[2] Alet, Ferran, et al. "Skillful joint probabilistic weather forecasting from marginals." arXiv preprint arXiv:2506.10772 (2025).

[3] Lang, Simon, et al. "AIFS-CRPS: ensemble forecasting using a model trained with a loss function based on the continuous ranked probability score." npj Artificial Intelligence 2.1 (2026): 18.

[4] Couairon, Guillaume, et al. "ArchesWeatherGen: Skillful and compute-efficient probabilistic weather forecasting with machine learning." Science Advances 12.17 (2026): eadx2372.

[5] Cachay, Salva Rühling, Duncan Watson-Parris, and Rose Yu. "U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster." arXiv preprint arXiv:2604.09041 (2026).

[6] Price, Ilan, et al. "Gencast: Diffusion-based ensemble forecasting for medium-range weather." arXiv preprint arXiv:2312.15796 (2023).

W4. The main CRPS improvements are reported without confidence intervals...

We thank the reviewer for this comment. ArchesWeatherGen is a strong 1.5° baseline, and in particular its results on the WeatherBench2 leaderboard have benefited from extensive engineering optimization. In Fig. 3 we include almost all representative ERA5 variables, four surface and five pressure-level variables, nine in total. Taking the 24-hour forecast as an example, Hyan improves on the pressure-level variables (z500, mslp) by about 2.5%, on the wind variables (u850, v850, u10, v10) by close to 5.5%, and on the temperature variable t850 by 1.11%, with a decrease only on t2m (-5.07%) and q850 (-1.71%). This indicates that the proposed method does offer a measurable advantage. On the other hand, under the inference-speed evaluation protocol of ArchesWeatherGen, our inference cost is only about 10%, which shows that Hyan achieves competitive or even better results at very low inference cost. We would argue that this already supports the conclusion of the paper.

W5. The experiments mainly evaluate aggregate forecasting metrics, leaving the model's ability...

We thank the reviewer for raising this point. We agree that the original evaluation did not cover this point. We therefore select eight representative typhoons from 2020, including Goni, the strongest typhoon of that year, and compare our model against IFS ENS with 40 ensemble members. The IFS ENS results are taken from WeatherBench2 and likewise use 40 members. The statistics are summarised below.

Lead time (h)24487296120
Ours60.1 ± 29.291.7 ± 29.3159.9 ± 49.1261.3 ± 94.0379.5 ± 105.0
IFS ENS85.9 ± 41.1144.5 ± 37.6227.0 ± 87.8323.7 ± 180.1623.3 ± 403.2

In addition, we use Chan-Hom as a representative case for visualisation analysis, as shown in Fig. 4 of the paper.

Reviewer_Mzg1OpenReview ↗

Reply to the authors' rebuttal

I appreciate the authors' clarifications and additional experiments. However, several concerns of mine still remain unresolved.

W1. The rebuttal does not sufficiently address the physical motivation and theoretical justification of the proposed perturbations. Comparisons with limitations of existing methods do not establish the validity of the proposed approach.

W2. The updated computational results are appreciated, but concerns regarding fair and consistent comparisons remain.

W3. Engineering design and choices are essential for bridging the gap between high-level methodological principles and practical deployment. While conceptual contributions may suffice for simpler or proof-of-concept problems, ensemble deep learning weather forecasting is a complex and highly competitive field attracting attention from both academia and industry. Systematic investigation of these choices, together with deeper technical design and theoretical justification, is necessary to establish the practical value and maturity of this work.

W4. The rebuttal does not provide statistical evidence to support the claimed improvements, particularly over strong baselines. Statistical evidence is very important to justify that the improvement achieved by a new work is really better than its previous counterparts, rather than biased experimental observations.

W5. The additional experiments are appreciated, but their limited scope does not yet establish the model's general reliability in predicting extreme events.

Overall, I believe the work requires more thorough investigation and stronger theoretical and empirical support to meet the expectations of this research community. I therefore maintain my original assessment and encourage the authors to further strengthen the paper before resubmission.

We thank the reviewer for the feedback. We would like to clarify a few points below.

W1. Our method is motivated precisely by the shortcomings of prior approaches, and its physical motivation rests on both intuition and empirical observation: perturbations should not be uniformly distributed across the global space, as prior methods assume, but should be correlated with the state. We did observe such a correlation in the autoregressive (AR) process, and the intensity-distribution visualization already present in the paper provides supporting evidence for it. In the latest revision, we have strengthened this argument in Figure 2.

W2. As explained in our earlier response, all evaluations follow the framework proposed by ArchesWeatherGen and UCast. The proposed Hyan method was pre-warmed up on a physical A100 GPU and evaluated as the average of 30 repeated measurements, which ensures that the comparison is fair and stable.

W3. As noted in our earlier response, we have conducted thorough ablations on the engineering choices underlying our core method, in particular on the source of randomness, and we compare against mainstream methods under fully identical conditions. The experimental settings were chosen so as to prioritize actual performance, so that the reported results reflect model capability under realistic conditions.

W4. In the revised manuscript we describe the CRPS performance as "competitive" because, given the substantially smaller parameter count and the significantly better inference time, an improvement of close to 1.5% is already of considerable value. For reference, the relative improvement of GenCast over ArchesWeatherGen is only about 3%. We would also like to note that training models for the WeatherBench2 leaderboard is very time-consuming. A single run consisting of training, fine-tuning, and full-year 2020 evaluation takes about a week, so evaluating 5 seeds alone would require more than a month. For this reason, multi-seed confidence intervals are rarely reported in this line of work, and we follow the same convention.

W5. Our method is not specialized for extreme weather, but aims at general probabilistic forecasting evaluation, which is why we adopt WeatherBench2 as the evaluation protocol. To our knowledge, WeatherBench2 does not prescribe any specific setting for extreme weather. In this line of work, extreme weather is typically evaluated through tropical cyclone track forecasting, and following the reviewer's suggestion we have added such an evaluation to the revision. Since tropical cyclone track forecasting is the standard proxy for extreme weather, we believe this addition provides a meaningful and representative assessment of model behavior under such conditions.

We hope these clarifications address the reviewer's concerns.

Reviewer_LVc5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hyan, an efficient ensemble weather forecasting model, and its deep-ensemble variant, Hyan DE. Hyan introduces state-dependent mixing coefficients at each noise injection layer to control the mixing of latent noise and latent features. These coefficients are predicted from the current latent features. In experiments on WeatherBench2, Hyan and Hyan DE show a good performance–efficiency balance compared with the baselines presented. Ablation experiments further analyze the effectiveness of state-adaptive modulation.

Critical strengths
  1. Clear motivation and a corresponding solution. Stochastic forecasting models may ignore noise and produce insufficient ensemble spread. Hyan addresses this issue by explicitly predicting local perturbation coefficients.
  2. Good results in the main experiments. Compared with models such as ArchesWeatherGen and GenCast, Hyan achieves a good performance–efficiency balance, demonstrating its practical value.
Critical weaknesses
  1. The existing ablation experiments need to be clarified and strengthened. Hyan uses two-stage training to improve forecasting and ensemble performance, but it is not clearly stated whether the dropout variant also underwent equivalent second-stage training, specifically to improve ensemble performance with dropout perturbations enabled. In addition, Static Gate not only removes state dependence but also has fewer parameters and less expressive capacity. Therefore, it is difficult to rule out the contribution of additional model capacity to Hyan's gains.
  2. As I understand it, Hyan's core novelty lies in its state-conditioned perturbation gating, but the current experiments do not distinguish the effects of the state-dependent coefficients on the noise from their effects on the latent state. Ablation experiments using z~=z+γ⊙ϵ\widetilde{z} = z + \gamma \odot \epsilon and z~=γ⊙z+ϵ\widetilde{z} =\gamma\odot z + \epsilon could be added.
Questions for authors
  1. Compared with Hyan (1x10), Hyan DE (4x10) changes both the number of models and the ensemble size, which confounds their respective contributions. Could the authors provide experimental results for Hyan (1x40)?
  2. If Hyan and Hyan DE were extended to a resolution of 0.25∘0.25^\circ, what would their approximate training and inference efficiency be? Would they retain their current efficiency advantage over GenCast?
  3. There are some inconsistencies in how the results are described in the paper. The caption of Figure 3 uses the phrase “substantially outperform,” although the performance gains over some baselines are limited. The term “competitive” used in Section 6 seems more accurate.
3 comments

We really appreciate the reviewer for the constructive comments and positive feedback on our paper.

W1. The existing ablation experiments need to be clarified and strengthened. Hyan uses two-stage training...

We apologize for the unclear presentation. The experimental protocol is as follows. We first train a backbone with no perturbation module at all. For each variant, the corresponding perturbation module is then attached with random initialisation, and the identical fine-tuning procedure is applied before the metrics are measured.

As for parameters, the AdaLN and Dropout variants add almost no extra parameters and are both at 86M, whereas the Static Gate and State-conditioned Gate variants are both at 100M. The additional 14M comes from the noise projection layer, a linear layer that maps the 14 spherical-harmonic noise fields to the latent dimension and takes no part in feature extraction. The exact parameter counts are as follows:

  • Global AdaLN (86.43M)
  • Feature Dropout (86.10M)
  • Static Gate (86.12M + 13.99M)
  • State-conditioned Gate (86.45M + 13.99M)

All of the above settings are now described in the paper (Section 6.1).

W2. As I understand it, Hyan's core novelty lies in its state-conditioned perturbation gating, but the...

Thank you for this suggestion. Rather than removing the coefficient outright, we first computed the expectation of γ\sqrt{\gamma} under γ∼U(0,1)\gamma\sim\mathcal{U}(0,1), which is 2/32/3, and substituted this constant value in the ablation. This preserves the mean scale of the perturbation, so that removing the state-dependent coefficient does not substantially shift the output distribution. The results are as follows.

ConfigurationMetrict2mz500q850u850Average
23⋅z+γ⊙ε\frac{2}{3}\cdot z+\sqrt{\gamma}\odot\varepsilonCRPS0.385661.45442.200.9896–
SSR0.88131.04220.90890.9867–
Perf. change−1.35%−47.69%+3.05%−1.87%−11.96%
1−γ⊙z+23⋅ε\sqrt{1-\gamma}\odot z+\frac{2}{3}\cdot\varepsilonCRPS0.386560.35441.960.9879–
SSR0.87681.04510.90710.9816–
Perf. change−7.43%−19.48%+4.93%+0.57%−5.35%
HyanCRPS0.385459.59442.590.9883–
SSR0.87781.05260.91300.9845–

The results show that perturbing the latent variable zz accounts for the larger share of the improvement, while the noise contributes less. Combining the two gives the best performance overall.

These results are now included in the ablation study in the paper (Section 6.2). We thank the reviewer for this helpful suggestion.

Q1. Compared with Hyan (1x10), Hyan DE (4x10) changes both the number of models...

We thank the reviewer for raising this point. Our previous evaluation did not control for ensemble size, which matters for RMSE. We have therefore re-evaluated both configurations, 1×40 and 4×10, both using 40 ensemble members. With this matched setting, RMSE improves markedly for standard variants, and the DE variant retains an advantage, though the gap is now smaller. The updated results are now reported in Table 6 and 7.

Q2. If Hyan and Hyan DE were extended to a resolution of (0.25^\circ), what would...

We thank the reviewer for this question. Hyan is faster because it removes the iterative sampling required by DPMs and instead uses a more efficient perturbation mechanism. Consequently, when Hyan is extended to 0.25°, its speed profile should resemble that of deterministic models such as FuXi and Pangu-Weather, in which a single forward pass completes a forecast, without the repeated sampling that GenCast requires. Hyan also adopts an efficient parameter design, with only 100M parameters in the current configuration, which keeps its parameter budget compact relative to ensemble-based models and gives it a further speed advantage. Taken together, we expect Hyan to retain this speed advantage when extended to 0.25°.

Q3. There are some inconsistencies in how the results are described in the paper....

We thank the reviewer for this correction. We agree that “competitive” is the more appropriate wording for the reported metrics, and we have updated it throughout the paper.

Thank you for the detailed response and the additional experiments. I particularly appreciate the new ablation studies and the comparison between Hyan (1×40) and Hyan DE (4×10), which directly address my main concerns. Overall, I believe my original concerns have been adequately addressed, and I maintain my positive assessment of the paper. Nevertheless, I share reviewer Mzg1's view that, although the proposed method is supported by empirical evidence, a stronger theoretical motivation or justification for the state-dependent perturbation mechanism would make the work more complete.

Additionally, I noticed a potential inconsistency in the authors' response to reviewer EkvN (W2.1). The simplified comparison with U-Cast appears to show a more substantial advantage for Hyan than the results presented in Figure 3 and Table 6. In the latter, the performance of Hyan and U-Cast appears relatively close. I suspect this discrepancy may be related to how the results are averaged across forecast lead times. A line plot showing CRPS as a function of forecast lead time for Hyan and U-Cast would also be helpful in understanding their relative performance.

For now, I will maintain my original score.

We are grateful to the reviewer for the effort devoted to our paper and for the constructive comments, which have helped us improve the manuscript.

Regarding the reviewer's suggestion on motivation and justification, we have revised the description of our method's motivation in the Related Work section and refined the presentation of the motivation in the overview figure (Figure 2). The basic premise of our method comes from intuition and empirical observation: perturbations should not be uniformly distributed across the global space, but should be correlated with the state. Indeed, we observed exactly such a correlation in the autoregressive (AR) process, and we have included this result in the latest revision to strengthen the motivation.

As for the inconsistency in UCast's performance, we would like to clarify that it arose because we initially used an older version of the dataset, in which the values at the missing entries of the SST variable differed. This resulted in lower evaluation metrics, which are the numbers in the table we attached to our reply. After the AC pointed this out, we re-examined our code, fixed the error, and updated the paper with the correct evaluation results. We also posted a note in the OpenReview discussion to declare the correction. We apologize for not updating the table in the earlier reply. Since the thread had already received responses, we chose to leave it unchanged to preserve the continuity of the discussion. We are also grateful to the reviewer for suggesting a plot of the average CRPS as a function of lead time, which we have added to Figure 9.

Reviewer_EkvNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission proposes a transformer based probabilistic weather forecasting model that generates ensembles on the fly during the forward pass via noise injection. A key aspect is that the layer-wise perturbations are made state-dependent through a dimensional bottleneck.

Critical strengths
  • The submission is well written and relatively easy to follow. This was a refreshing read among the other submissions. It really felt like this paper was written largely by humans with the intent of actually explaining things clearly to other humans.
  • The method is fairly simple and based on intuition or heuristics drawing heavily from existing work. This is all made clear and the authors do not try to hide that these are engineering choices. It makes clear that the submission intends to be carried by empirical results. As far as I can tell, this is done well: solid baselines (though possibly incomplete, see below) and a reasonably complex benchmark paired with multiple relevant metrics does largely appear to support the claims that the proposed method is indeed competitive.
Critical weaknesses
  • In terms of novelty and scope the main contribution appears to be the layer-wise state-conditioned adaptive noise injection via the learned mixing coefficient. Generally, injecting perturbations throughout deep learning models for stochastic ensemble generation have been used quite extensively (from simple dropout, to spatially and temporally correlated fields in FourCastNet 3, etc). In the end, this mostly serves the purpose of creating diverse ensembles in the probabilistic fine-tuning stage. This section (Stage 2) is a bit obscuring to me. As I understand, the loss ultimately is exactly a weighted fair CRPS loss, correct? I believe this should just be stated directly instead of a handwavy derivation from an inverse Lipschitz criterion. Stating inverse Lipschitz, realizing that the RHS is constant and then especially the (big?) leap to enforce that criterion by maximizing the LHS (much more than satisfying a bound) are too many unnecessary leaps to end up at a fairly well understood weighted CRPS loss. In particular, the training ensemble size M=3 (to be honest, was a bit surprised by this, I somehow assumed M would be much larger in practice), I remember from AIFS-CRPS that there can be configurations (all equal but one) that have zero fair CRPS, but still are not really desirable. They mitigate this by "almost fair CRPS". It appears this is not an issue the authors ran into during the experiments, but may still be worth investigating.

  • Some questions about baselines:

  1. Would it be possible to include U-Cast [1]. I understand it is pretty recent, but it is a comparably simple method that also relies on deterministic pre-training with probabilistic training using CRPS as signal aiming at efficient ensemble generation. From their results, they appear to claim similar CRPS improvement together with high efficiency compared to some of the same baselines.
  2. The authors mention in l.85 that some deterministic methdos have "non-deterministic formulations", but those are not followed up on. In particular, as far as I remember, OmniCast is a probabilistic diffusion model providing probabilistic forecasts? Similarly, FuXi also provides (simple) ensembles. Why does it not make sense to include those?
  • Ultimately, the empirical results are good, but I feel they don't fully support the sweeping claims. For example, in figure 3, I struggled to verify the claim that balines are "substantially outperformed" given that the Hyan and Hyan DE lines are completely sandwiched between the baselines and sometimes dominated not only by GenCast but also by ArchesWeatherGen (would be interesting to see U-Cast here). The abstract also reads as if Hyan essentially obliterates all existing methods, when the results are really a bit more nuanced and mixed. I don't expect substantial superiority across all axes, but the claims should perhaps be a bit more nuanced then in some places (later there's also a "consistently better throughout the forecast period", which is not fully supported unconditionally).

  • Side remark: The submission lacks the mandatory AI use statement

[1] Cachay, S. R., Watson-Parris, D., & Yu, R. (2026). U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster. arXiv preprint arXiv:2604.09041.

Questions for authors
  • Lots of notation is dropped quite quickly from Section 4.1 onwards without exlpaining all symbols. In eqs. (3)-(6) alone are almost a handful of symbols that are not mentioned in the text. It would have helped me a lot if there was a short explanation of each symbol together with what kind of mathematical object it is. Similarly, a few choices like the activation funciton \kappa and some other more detailed choices remain unclear or without motivation.

  • In the main figures, it would be good to re-state clearly what "Improvement" means, i.e., what exactly is plotted on the y axis.

  • Since Hyan DE is using 40 ensemble members and Hyan only 10, can we disentangle which improvements of Hyan DE over Hyan in averaged metrics simply come from averaging larger ensembles? Is a comparison with Hyan using 40 members possible as well?

5 comments

We really appreciate the reviewer for the constructive comments and positive feedback on our paper.

W1. In terms of novelty and scope the main contribution appears to be the layer-wise state-conditioned adaptive noise injection via...

We thank the reviewer for pointing out the issues in Section (Stage 2), and we apologize for any confusion caused by the original presentation of the loss function. The derivation related to the inverse Lipschitz property in Section (Stage 2) is indeed not a core contribution of this paper, and it can easily mislead the reader. To improve this, we have substantially simplified the presentation at that point, retaining only our most basic understanding — namely, preventing the collapse of the forecast distribution.

W2.1. Would it be possible to include U-Cast....

UCast did not release its official forecasts on WeatherBench2, so we reproduced them ourselves. To this end, we used the official code (github.com/Rose-STL-Lab/u-cast) with the released checkpoints (ucast_de3.ckpt), and computed all scores with the repository's own metric scripts. M=40, initialized at 00 and 12 UTC for the whole of 2020. Simplified results are summarized below. Values are averaged over forecast days 1–5. Full results are reported in Fig. 3 and Table 6.

VariableUCast CRPSHyan CRPSUCast SSRHyan SSR
t2m0.4800.3850.7180.912
u100.7060.6280.8631.045
z50067.2960.280.8061.104
q850470.94441.920.8870.948

W2.2. ...Why does it not make sense to include those?

We would like to make a correction here: OmniCast should be categorized as a probabilistic forecasting model, and placing it among the deterministic models was a mistake, which we have now corrected.

The main reason for not including these baselines is that the baselines we do compare against are already strong. All of them are established WeatherBench2 baselines whose scores are already publicly reported as part of the benchmark itself and in the papers of the respective methods. They have also been extensively engineered and validated against the official benchmark, and we evaluate our model under the same WeatherBench2 protocol and on the same data, so no additional reproduction is needed for a like-for-like comparison. In addition, as reported in the FuXi and OmniCast papers themselves, these models may not yet show a clear advantage over IFS ENS, so reproducing them would add limited value to the comparison.

W2.1. & Q2.

We thank the reviewer for raising this point and apologize for the confusion. Following the suggestion, we have redrawn the figures. We have also switched the reference baseline from IFS ENS to ArchesWG, which makes the differences between the models easier to see. We hope these revisions address the reviewer's concern and present the comparison more clearly. CRPS is now shown in Fig. 3 of the main text, and RMSE has been moved to Appendix G.

Q3. Since Hyan DE is using 40 ensemble members and Hyan only 10...

We thank the reviewer for this careful observation. Our previous evaluation focused on CRPS, a metric that is largely insensitive to ensemble size. RMSE, by contrast, does depend on the number of members, and the two variants were not evaluated with the same ensemble size, so the RMSE comparison was not as strictly controlled as the CRPS one. We have therefore re-evaluated both the standard and the DE variants with 40 members. With matched ensemble sizes the RMSE gap narrows, and the DE variant still improves over the standard variant overall. Our conclusions remain unchanged, and the updated numbers are reported in Table 6 and 7.

W2.2. & Q1.

We thank the reviewer for pointing this out. We have now added the AI use statement and the missing notation definitions.

Large language models (LLMs) were used to assist with writing utility scripts for plotting and statistical analysis, and to refine the wording of the manuscript. They were not used to modify the core model implementation or generate any figures. All research questions, methodological choices, experimental decisions, results, and conclusions were determined and reviewed by the authors, who take full responsibility for the content of this paper.

Dear Authors,

Thank you for your very constructive engagement with the reviewers comments. To further support the reviewer and me in our assessment of your work, I would be grateful if you could also help clarify one thing in your answer: Is there a clear reason that you could not use the U-cast evaluation numbers from the original paper, as they seem to be for the same set of initial conditions in 2020? What is the reason for the large discrepancy between your U-cast CRPS results and the ones in the original paper (for the non-DE version of U-cast)?

We thank the Area Chair for this additional comment. For the UCast metrics, we used the official weights and code to obtain complete and detailed scores, which is standard practice for such comparisons. As for the low scores, a careful investigation shows that they originate from the WeatherBench2 dataset.

Specifically, we used a slightly older WeatherBench2 zarr, 1959-2022-6h-240x121_equiangular_with_poles_conservative.zarr, whereas UCast defaults to the newer 1959-2023_01_10-6h-240x121_equiangular_with_poles_conservative.zarr. The two versions do not differ in data content or format, so we initially ran UCast on the older version we already had, in place of its default dataset. After closer inspection, we found that the two versions handle missing values in the SST variable slightly differently, which triggers unexpected behaviour in UCast's official evaluation code. About 10% of the SST entries may differ from what the model saw during training, which degrades the resulting scores. We have now fixed the unexpected behaviour of the code on SST and obtained the correct scores, which have been updated in Fig. 1, Fig. 3 and Tables 6 and 7.

We would also like to note that the comparison with UCast has little bearing on the conclusions of the paper. Much of UCast's strong performance can be attributed to its much larger parameter count (approximately 900M), compared with 100M for Hyan. The comparison that holds the backbone fixed is already given by the ablation in Sec. 6.1, where the proposed method shows an advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.