- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes PrecMamba, a two-stage precipitation nowcasting model. Stage one is a deterministic backbone: a Hilbert-curve-serialised, temporally hierarchical bidirectional Mamba encoder with untied per-lead decoders. It is trained with a multi-quantile pinball loss so that it learns more than the conditional mean of the heavy-tailed intensity distribution. Stage two is a conditional flow-matching U-Net that models the scaled residual around the median forecast, conditioned on the frozen encoder features and the deterministic forecast, and produces ensemble members in about 10 Euler steps. On SEVIR, MeteoNet, Shanghai Radar and CIKM the method reports the best CSI and SSIM, gains at the two highest SEVIR thresholds, and more than an order of magnitude faster residual sampling than DiffCast's diffusion generator. Ablations cover the loss, backbone/generator pairings, scan order, ensemble size and number of Euler steps.
Critical strengths
- Consistent gains on four benchmarks, especially for extreme precipitation. PrecMamba has the best CSI and SSIM on all four datasets, with the largest margins at the neighbourhood scale (CSI-pool16). On SEVIR, CSI at the highest threshold is roughly twice DiffCast's, which directly addresses the paper's stated goal of forecasting high-impact events.
- The motivation is simple and supported by ablations. Training the deterministic stage with a pinball loss beyond the conditional mean beats both MSE and threshold-weighted MSE by a clear margin (CSI 0.3445 vs 0.2775/0.3115; CSI-pool16 0.609 vs 0.295/0.410). This gives a useful and easily transferable insight for residual-based nowcasting models.
- Large practical efficiency gain for ensemble forecasting. Replacing diffusion with conditional flow matching cuts residual sampling from about 1.23 s to about 0.07 s per forecast, improves accuracy on both backbones, and reaches near-saturated quality in about 10 steps. This makes multi-member ensembles feasible for operational use.
Critical weaknesses
- The comparison protocol is weak and statistical robustness is limited. Most baseline numbers are copied from two different papers, and DiffCast is reported with a different best backbone on each dataset. All results come from a single stochastic sample with no seeds, variance or significance testing. Some margins are small (e.g., MeteoNet CSI 0.3941 vs 0.3892), and HSS is behind DuoCast on SEVIR and MeteoNet, so the strength of the SOTA claim depends on how reliable these comparisons are.
- The probabilistic claims and the mechanism are not fully validated. Ensembles are a central selling point, yet there are no probabilistic metrics such as CRPS, spread–skill ratio, rank histograms or reliability diagrams to show the ensemble is calibrated. The backbone alone also underperforms SimVP, and the paper attributes the benefit of the upper-quantile terms to indirect effects through the encoder features without an ablation that isolates this. As a result, it is unclear how much of the gain comes from the quantile supervision versus the flow generator itself.
Questions for authors
See the weaknesses part.
3 comments
Reply to Reviewer yRdL
We thank the reviewer for their positive assessment and their follow-up comment, and we are glad that they found the gains consistent and the efficiency useful for ensemble forecasting. We address both weaknesses below.
W1: Comparison protocol and statistical robustness
Baseline sources. Both sources follow the same DiffCast protocol, with the same datasets, preprocessing, splits and evaluation code. For DiffCast, we report its best backbone on each dataset, which is the most favourable choice for this baseline. We also ran the released DiffCast checkpoint on the same test windows as our model (App. A.1, Tab. 3).
Single sample. We use a single sampled member to follow the DiffCast protocol, so that our numbers are comparable with the published results. To show the variability, we report the standard deviation of CSI over 10 independent samples, next to the margin over the strongest baseline in Tab. 1:
| Dataset | Std of CSI | Margin over strongest baseline |
|---|---|---|
| SEVIR | 0.0025 | 0.0070 (DuoCast) |
| MeteoNet | 0.0023 | 0.0049 (DuoCast) |
| Shanghai Radar | 0.0033 | 0.0239 (DuoCast) |
| CIKM | 0.0020 | 0.0087 (AlphaPre) |
The margin exceeds the standard deviation on all four datasets. On MeteoNet, where the margin is smallest, it is about twice the standard deviation.
Small margins and HSS. We agree that the margin on MeteoNet is small and that DuoCast keeps a small advantage in HSS on SEVIR and MeteoNet, as stated in Sec. 4.2. We will revise the text so that the SOTA claim refers to CSI only.
W2: Probabilistic evaluation and mechanism
Probabilistic metrics. We now report CRPS and the spread–skill ratio of an 8-member ensemble (CRPS in the native units of each dataset):
| Dataset | CRPS | Spread–skill ratio |
|---|---|---|
| SEVIR | 6.130 | 0.575 |
| MeteoNet | 0.772 | 0.538 |
| Shanghai Radar | 1.486 | 0.509 |
| CIKM | 2.846 | 0.455 |
The spread–skill ratio is below 1, so the ensemble is under-dispersed. A likely reason is that the generator models only the residual around a fixed deterministic forecast, so it does not represent the uncertainty of μ itself. We will report this in the paper as a limitation. We are also computing these metrics for the released DiffCast checkpoint on SEVIR, and we will post them in a follow-up comment as soon as they are available.
Quantile supervision versus generator. With the MSE-trained backbone without the generator, we can now give the full 2 × 2 comparison on SEVIR:
| Objective | No generator | Flow generator |
|---|---|---|
| MSE | 0.2535 | 0.2775 |
| Pinball | 0.2830 | 0.3445 |
The pinball objective adds 0.0295 CSI to the backbone alone, but 0.0670 to the full model. The generator adds 0.0240 to the MSE backbone, but 0.0615 to the pinball backbone. So most of the benefit of the quantile supervision appears through the generator, which is consistent with our explanation that it acts mainly through the shared encoder features.
Backbone alone below SimVP. We agree that our backbone alone is a weaker deterministic forecaster than SimVP on SEVIR, since, with the same MSE objective, it reaches 0.2535 against 0.3108. Its value appears in combination with the pinball loss and the generator, since, with the same objective and generator, SimVP reaches 0.3236 on SEVIR, against 0.3445 for PrecMamba.
post-rebuttal
This paper proposes PrecMamba, a two-stage precipitation nowcasting framework that combines a Mamba backbone trained with multi-quantile supervision and a conditional flow-matching residual generator to improve heavy-precipitation forecasts and enable efficient ensemble sampling.
The strengths of this paper includes: (1) Consistent gains on four benchmarks, especially for extreme precipitation; (2) Ablations supporting the motivation of using residual-based nowcasting models; (3) Large practical efficiency gain for ensemble forecasting.
However, there exists some weaknesses:
-
The comparison with baselines are noisy, since all baseline numbers are copied from prior papers rather than re-run under a consistent experimental setup. Moreover, some small margin gains reduce the confidence of the SOTA claim. During the rebuttal, the authors clarify that the baseline sources followed the same DiffCast protocol, and agree to narrow their SOTA claim to CSI.
-
The probabilistic claims and the mechanism are not fully validated. During the rebuttal, the authors provide additional experiment results such as CRPS and spread-skill ratios, which address my concerns.
Overall, after carefully reading the authors' rebuttal and other reviewers' comments, I think my concerns have been well addressed. Although the work still has some limitations, I think this paper meets the standard of ICLR. Nevertheless, as I am not deeply familiar with this specific area, I leave the final decision to AC.
Dear authors,
I have carefully read the comments from other reviewers, and I remain positive about this work at this stage. Looking forward to your response to the concerns I raised.
Best