- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes overfitting and performance degradation in existing CRPS-based probabilistic weather forecasters when directly predicting distant futures with high uncertainty. To address this, it jointly learns auxiliary conditional denoising tasks that provide a corrupted version of the future state alongside the forecast context. The intuition is that learning easier tasks with partial future information through shared parameters can benefit the original context-only forecasting task. Building on Distributional Diffusion Models (DDMs), the method learns conditional future distributions across noise levels using a CRPS-based objective, adding the corrupted future and its noise level as conditioning inputs to existing forecasters. At inference, it uses the fully corrupted endpoint (), which contains no future information, retaining one model evaluation per ensemble member per forecast step (NFE=1). Under matched architectures and training settings, the method reduces nCRPS for direct three-day forecasting by 9.4% for FGN and 8.1% for FCN3. In the global MOSAIC experiment with 84 forecast channels, it improves CRPS in 93.7% of the 840 channel–lead-time pairs over a ten-day rollout.
Critical strengths
-
A coherent problem–method–evaluation narrative. The paper identifies difficulties in direct long-horizon forecasting, provides evidence of overfitting in this setting, and introduces a plausible intervention through auxiliary denoising tasks. The experiments then demonstrate that the intervention mitigates the observed degradation. This is a reasonable and well-organized narrative, although the causal connections require stronger support, as discussed below.
-
Strong empirical performance against matched baselines. The proposed training objective delivers substantial gains over standard CRPS training in the more challenging long-horizon direct forecasting tasks. Improvements also extend to larger global forecasting models while preserving single-evaluation ensemble generation.
-
Extensive supporting analyses. The paper provides analyses of the auxiliary-task noise range, forecast horizon, sampling steps, autoregressive rollouts, ensemble calibration, distribution shift, and tropical cyclone cases. These experiments help characterize both the benefits and the conditions under which improvements are limited.
Critical weaknesses
-
The title and method naming do not clearly communicate the distinctive contribution. The central design choice is to use auxiliary conditional denoising tasks to improve context-only forecasting. However, neither the title nor the DDM label used in the experiments makes this contribution explicit. Since DDMs and proper-scoring-rule objectives originate in prior work, clearer terminology would help distinguish the proposed auxiliary-training strategy from the underlying framework.
-
The proposed explanation for the gains needs more rigorous validation. The paper connects difficult long-horizon forecasting to overfitting, and motivates auxiliary tasks as a source of regularization that alleviates this problem. While the experiments support the effectiveness of the intervention, they do not firmly establish these causal links. In particular, why does overfitting become more pronounced at longer direct forecast horizons, and through what specific mechanism does auxiliary denoising regularize the shared predictor? Given the simplicity of the method, a more rigorous account of why it works would substantially strengthen the contribution. Relevant prior theory on auxiliary-task regularization, together with controlled comparisons against conventional regularization or alternative auxiliary tasks, could help substantiate this interpretation.
-
The scope of empirical validation is narrower than the apparent applicability of the method. The proposed auxiliary-training strategy does not appear inherently specific to weather forecasting: it could potentially apply to other conditional probabilistic prediction tasks. Evaluating it in at least one additional domain would help establish whether the benefits reflect a broader learning principle or depend on properties of the weather datasets and architectures studied here. I recognize that the authors contrained the scope for this paper to weather forecast which is the one of the important forecasting task. However, the method or the motivation is not the problem only for weather case. Given that overfitting motivation and regularization method could be a problem in any forecasting problem, I believe that this paper should have some experimental results for other domains.
Questions for authors
See weaknesses section.
5 comments
(CW2) In particular, (1) why does overfitting become more pronounced at longer direct forecast horizons and (2) what specific mechanism does auxiliary denoising regularize the shared predictor?
-
We first clarify the weather-specific context motivating long-horizon direct forecasting. In other applications, such as robot manipulation [1] or video generation, a common goal is to produce a coherent sequence of actions or frames at fine temporal resolution. In weather forecasting, on the other hand, there is also a strong demand for predictions at longer lead times (e.g., medium-range weather forecasts), even when densely spaced intermediate states are not needed. Direct forecasting addresses this need by predicting a distant future state without sequential rollout, avoiding repeated model evaluations and the accumulation of rollout errors.
-
Why does overfitting become more pronounced at longer direct forecast horizons? Direct forecasting requires learning a future distribution across a long interval in a single step, with uncertainty increasing due to chaotic atmospheric dynamics [2]. Yet only one realized future is available per training context. With finite data, a flexible model can lower its empirical CRPS by fitting these individual realizations without learning a conditional distribution that generalizes. We hypothesize that this tendency becomes more pronounced at longer horizons, where greater predictive uncertainty makes realization-specific variation more substantial, consistent with the stronger overfitting observed in Figure 2.
-
Through what mechanism does auxiliary denoising regularize the shared predictor? Our auxiliary tasks provide partial future information during training: the model predicts the same future from the context and a corrupted future at different noise levels. At low noise levels, realization-specific details can be recovered from rather than fitted through . As the noise increases, becomes less informative, and the model must rely more on . Crucially, these tasks share the same target and network parameters. We hypothesize that this shared training encourages the model to use for generalizable predictive structure and for realization-specific details, reducing its tendency to overfit to the latter through at the context-only endpoint .
-
The training dynamics of D$^2$Cast in Figure 14 support this mechanism. At longer horizons (3d and 5d), D$^2$Cast keeps a higher training ensemble RMSE at than CRPS training, yet its validation RMSE falls below even the best value of CRPS training. This indicates that the lower training error of CRPS training comes from fitting the realization-specific part, which does not generalize to unseen data. D$^2$Cast instead represents this part as ensemble spread, which remains close to its validation RMSE rather than collapsing. At shorter horizons (12h and 1d), where this part is small, the two objectives behave nearly identically. We leave a formal analysis of this mechanism to future work and will add this discussion, including Figure 14, to the revised manuscript.
-
We also conducted controlled comparisons against conventional regularization and an alternative auxiliary task; please see our response to Q1 of Reviewer BeY1 for details. Across the four tested weight-decay/dropout configurations, conventional regularization provides only marginal gains and does not close the gap to D$^2$Cast, particularly at longer horizons.
(CW3) Evaluating it in at least one additional domain would help establish …
-
To address the reviewer's suggestion to evaluate our formulation beyond weather forecasting, we additionally evaluate it on the 2D incompressible Navier–Stokes dataset, a standard benchmark for learning dynamical systems in scientific machine learning.
-
Using the Navier–Stokes dataset from [3], we compare standard CRPS training and D$^2$Cast with the same U-Net backbone and matched training settings, directly predicting from for . We use 1000, 200, and 200 trajectories for training, validation, and testing, respectively. The table below reports test fair CRPS and ensemble-mean relative L2 error (10 members, NFE = 1), with denoting the relative change from the CRPS baseline.
| τ | Method | fair CRPS ↓ | rel. L2 (ens. mean) ↓ |
|---|---|---|---|
| 1 | CRPS | 0.0071 | 0.0147 |
| Ours | 0.0073 | 0.0150 | |
| Δ | +2.8% | +2.0% | |
| 2 | CRPS | 0.0278 | 0.0470 |
| Ours | 0.0272 | 0.0454 | |
| Δ | -2.2% | -3.4% | |
| 4 | CRPS | 0.0625 | 0.1061 |
| Ours | 0.0593 | 0.1000 | |
| Δ | –5.1% | –5.7% | |
| 8 | CRPS | 0.1236 | 0.2026 |
| Ours | 0.1148 | 0.1885 | |
| Δ | −7.2% | −7.0% |
- The result shows that D$^2$Cast slightly underperforms the CRPS baseline at the shortest stride but shows clear improvements at longer strides, consistent with the stronger gains at longer horizons in our weather experiments. Together with the improvements across diverse architectures (Table 14), these results suggest that the benefits of our formulation may extend beyond the specific datasets and architectures studied in weather forecasting, particularly under higher forecast uncertainty.
[1] De Bortoli et al., Distributional Diffusion Models with Scoring Rules, ICML 2025.
[2] Lorenz, The predictability of a flow which possesses many scales of motion, Tellus, 1969.
[3] Li et al., Fourier Neural Operator for Parametric Partial Differential Equations, ICLR 2021.
We sincerely thank the reviewer for their careful review and feedback. We are pleased that the reviewer found our paper to follow a “coherent” and “well-organized” narrative. We are also grateful that the reviewer acknowledged the “strong empirical performance” against matched baselines. Furthermore, we appreciate the reviewer’s recognition of our “extensive supporting analyses”. We appreciate the reviewer’s suggestions for further strengthening the paper and address each concern in detail below.
We appreciate the reviewer's suggestion to clarify our distinctive contribution relative to the prior work [1] and agree that clearer terminology would help distinguish our forecasting-oriented approach from the original DDM framework.
The main distinction from DDM lies in how conditional denoising distributions are used. DDM learns them for use in a reverse sampling process. Whereas our goal is to extend CRPS-based context-only weather forecasting through distributional diffusion, thereby intermediate conditional denoising tasks serve as auxiliary learning signals for the context-only forecasting at t=1 without being used at one-step inference.
In this respect, we believe that our current title reflects this methodological formulation. However, we agree with the reviewer that referring to our approach simply as “DDM” does not clearly distinguish our forecasting-oriented contribution from the original framework. We will therefore introduce the method name D$^2$Cast (Distributional Diffusion for Weather Forecasting) and update the manuscript.
To avoid confusing the other reviewers during the ongoing rebuttal period, we will revise the manuscript to reflect the new method name as soon as the discussion concludes.