- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors target the task of precipitation nowcasting, a well-known problem in meteorology which has been addressed in multiple ways over the years by the deep-learning community. While the problem is well-known, the authors address a number of issues with the existing solutions:
- proposal of forecast-sample the correction-calibrate pipeline
- availability conditioning by distinguishing between no event and no record of event
The authors highlight higher performance on benchmarks than SOTA.
Critical strengths
Clarity: The paper is clearly written, outlining the architecture and the experiments in a reproducible way
Quality: A number of experiments support the claims of superior performance of the method; it is especially important to highlight that the use of availability masks for handling missing data is shown, through the ablation studies, to be beneficial for the performance
Soundness: The approach is sound, and is grounded in existing body of literature on precipitation nowcasting
Originality: the way the work treats the missing data addresses a research gap and is a valuable addition to the existing methodology.
Critical weaknesses
Concerns about originality: there is an issue that this work seems to frame the approach as the novel view of the precipitation nowcasting as forecast-refine-calibrate pipeline. However, it seems to apply to multiple works, such as DiffCast, that decompose the forecasting into deterministic forecast, stochastic refinement, with subsequent calibration at the deployment time. Therefore, the key point would be to emphasise why this is a substantial step beyond works such as DiffCast.
Significance: The authors state that "Future work could relax the restriction of ensemble corrections to variations around the backbone forecast and expand the input area to provide additional spatial context, especially at longer horizons." However, these limitations should be more thoroughly described, opening the door for future research. Not sure it is clear from this text why the relaxation of this restriction and expansion of input area is technically challenging (which I assume is implied).
Questions for authors
I think the whole set of questions would be about clarifications of concerns above. If the authors can give a persuasive answer why (1) the forecast-refine-calibrate pipeline is substantially different, and (2) address concerns on the significance, that would address current concerns.
3 comments
In terms of my comments, the following substantial concern remains: it seems to me that the argument goes that a number of architectural decisions help outperform existing methods. An arguably better argument you may have would be to say what are the fundamental limitations of the current approaches which make it a principally different approach from the current works that separate a deterministic forecast from a stochastic correction.
Now the abstract says: "Existing approaches struggle to leverage diverse observation sources, to remain robust when sources are unavailable, and to efficiently generate ensemble forecasts. RainOps addresses these challenges with an efficient framework that separates deterministic forecasting from stochastic refinement." So, the argument goes that the framework is efficient, as opposed to proposing a step change. I would like the authors to say exactly why is it a step change, addressing the gap in the literature, and not four separate architectural improvements cited in the rebuttal.
Neither does related work contrast this work against other works that separate deterministic forecasts and stochastic corrections...
The limitations paragraph answers my question, thank you! I have also checked the responses to the other reviewers, I believe they mostly address the gap in the experimental design.
Thank you for the follow-up, and for reading the other responses as well. We're glad the limitations paragraph answered your question. We agree that the key question is what current methods that separate a deterministic forecast from a stochastic correction fundamentally cannot do. We see two limitations.
-
The ensemble is never trained. In DiffCast, CasCast and PercpCast, the stochastic stage is a generative model trained one sample at a time, to make each sample look realistic. An ensemble only appears at inference, when the model is sampled repeatedly. Nothing in training asks the members, taken together, to cover the observed outcome. Good spread therefore depends on sampling the generative model accurately, which takes many steps per member (1,000 network calls for one DiffCast forecast). PercpCast avoids this cost by giving up the ensemble altogether. RainOps instead trains the ensemble itself. Members are generated together during training and scored jointly with a proper scoring rule (fair CRPS), which directly rewards members that cover the observed outcome without drifting away from it. Because the spread is learned rather than recovered through sampling, two refinement steps are enough. In a controlled comparison (Appendix G.1), we kept everything else fixed and replaced only this training: the per-sample refiner gives blurrier members and needs 12–50× more steps to clearly improve CRPS.
-
Each model is tied to one set of inputs. Current methods assume that every input source is present at forecast time. If a satellite or lightning source is missing, the options are to train another model for that configuration, repeating the expensive generative training, or to feed in data the model was never trained to handle. With k sources, there are 2^k possible configurations. In RainOps, which sources are available is an input to the deterministic backbone, and the stochastic refiner does not depend on it at all. A single trained model therefore serves every configuration. In Appendix F, one RainOps model evaluated in all eight availability states improves with every added source and performs comparably to specialists trained for each state, without any retraining.
Together, these two differences give one model that produces an ensemble from whatever observations are available. A 10-member ensemble takes 146 ms, 13.5% longer than a single deterministic PercpCast forecast.
We thank Reviewer zkiH for their feedback and for recognizing the clarity of the paper, the soundness of the approach, and the value of our treatment of missing data. Below, we address the two questions raised.
(1) Why RainOps is substantially different from DiffCast. DiffCast and RainOps both separate a deterministic forecast from a stochastic correction, but they differ in four ways.
To isolate the refiner, we also built a DiffCast-style variant of RainOps (new Appendix G.1). It keeps all our other contributions, namely the same frozen multi-source, availability-aware backbone, the same inputs and conditioning, the quantile calibration, and the 10-member ensemble, and replaces only our flow refiner with a residual diffusion refiner of the same U-Net. Its members are clearly blurrier (LPIPS 0.30–0.35 vs. 0.245), and it needs 12–50× more steps to clearly improve CRPS.
(2) Significance and limitations. We added an extended Limitations paragraph (Section 5) that explains why these extensions are not straightforward. Ensemble members are corrections around the backbone forecast. This is exactly what makes two-step sampling possible, but it also means that members cannot move a misplaced storm (Appendix G.2). Sampling such displacements requires starting from noise with many more steps, which brings back DiffCast-like cost. At long lead times, rain entering from outside the domain cannot be predicted from the inputs, and expanding the input area is limited by SEVIR's fixed 384 km patches and increases training cost. Perturbing the backbone's latent state and training on larger domains are promising directions for future work.