- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the issue of extreme wind speed prediction and proposes a loss function called "Station-Calibrated Asymmetric Loss" (SCAL), which is combined with the spatio-temporal Transformer method to construct a complete prediction framework. This framework converts the large-scale ERA5 meteorological field data from the past 24 hours into data representations specific to each site, thereby predicting the wind speed for 18 stations under the Korea Meteorological Administration's jurisdiction in the next 24 hours. In the experiments, a baseline model was constructed by changing only the loss function while keeping the main network and training settings unchanged, thereby verifying the effectiveness of the proposed SCAL.
Critical strengths
- The author evaluated the model using multiple comprehensive indicators, including nMAE-ext, bias_ext, underreporting rate (under rate), POD, SEDI, etc.
- This paper focuses on the "systematic bias" issue in extreme weather forecasting; this is a problem of practical significance.
Critical weaknesses
- The primary innovation emphasized by the authors—the design of a loss function called SCAL—essentially amounts to introducing a position-specific calibration coefficient into the "extreme underestimation" mechanism of the existing EPL method. Regarding the architecture, the authors employ a decoupled modeling approach, utilizing independent spatial and temporal self-attention mechanisms; however, given the extensive existing research and established findings in this field, this approach is overly simplistic. Furthermore, SCAL lacks both a theoretical foundation and sufficient empirical validation to demonstrate its effectiveness.
- The main experiments establish a baseline solely by modifying the loss function, without comparing the approach against other relevant methods. The scope of the experiments is limited, covering only a single dataset, which is insufficient to demonstrate the method's generalization capability and robustness. Moreover, Table 1 fails to show that SCAL performs well; on the contrary, it indicates that SCAL actually leads to an increase in both overall error and extreme error.
Questions for authors
Q1. Are there any relevant experiments or other evidence indicating that the SCAL loss function (as well as the loss functions used in other backbone networks) is not only applicable to the architecture proposed by the authors but also can bring benefits to it? Q2. The ablation experiments show that removing the "site-specific representation path" will cause a deviation of "under-prediction" in the regression prediction. Does this mean that if the backbone network cannot identify site-specificity at the feature level and only relies on the penalty terms in the loss function, the model will be unable to learn the physical mapping relationship necessary to correct the deviation?
4 comments
Response to Reviewer m29S (2/2)
Q2 — Without station-specific features, can the loss alone correct the bias? Partly. Removing the station adapter, or the whole station representation path (station-aware cross-attention and adapter, after which the decoder query is identical for all stations), leaves the tail error and direction not significantly different from those of the full model (nMAE_ext 0.784 and 0.808 vs 0.794; under-rate 0.520 and 0.562 vs 0.541; with MSE + SCAL, 0.740 and 0.755 vs 0.734), and the ConvLSTM result (Q1), which has no station-specific path in the encoder, points the same way. The directional correction therefore does not require these two components, and we correct the submission’s statement that these components are required for it. However, the decoder still reads the encoder tokens at each station’s coordinates, so station location information remains available, and these variants are not an independent test without station information. We no longer claim a performance benefit for these components: at the selected checkpoints whole-range error differs between variants (0.884 and 0.778 vs 0.785), but variants checkpointed earlier, when κ is larger, have higher nMAE_all, and at the last epoch before early stopping all three lie within 0.68–0.69 on test. An attention analysis in the appendix shows that the cross-attention reads upstream areas (the northwestern continent and the southern sea) at up to about 2.7 times the uniform weight; we give this as a description of behavior, not as evidence that the component is needed.
Thank you for your thorough response. I have carefully read all of your comments, but we believe that several concerns remain unresolved: $ \textbf{1.}$ The paper does not introduce any innovative modules or methods, and merely exploring different loss functions is insufficient to establish methodological novelty. $ \textbf{2.}$ The experiments are conducted on only one small-scale dataset, without sufficient evidence of generalization across multiple datasets. $ \textbf{3.}$ The experimental evaluation lacks comparisons with state-of-the-art (SOTA) methods, further limiting the credibility of the paper's claims regarding its effectiveness.
Based on these concerns, I maintain my original evaluation and score.
Thank you for your follow-up
Thank you for taking the time to carefully review our responses and revised manuscript. We appreciate your thoughtful feedback and acknowledge that the concerns you raised regarding methodological novelty, generalization across datasets, and comparisons with state-of-the-art methods remain important limitations of our work.
Your comments have helped us better understand the limitations of the current study and identify directions for further improvement. Thank you again for your time and constructive feedback.
Response to Reviewer m29S (1/2)
Thank you for the expert review. We address the reading of Table 1 first, then novelty, scope and your two questions. Note that we found and fixed a 9-hour misalignment between observations and ERA5 inputs and retrained every model; all numbers below come from the corrected pipeline, and the General Response lists what the correction changed.
W2 — “SCAL increases both overall and extreme error in Table 1.” SCAL lowers extreme-range error and raises whole-range error. This was true in the submission (nMAE_ext 0.837 for EPL vs 0.721 for SCAL) and holds after correction: nMAE_ext falls from 0.988 (EPL) and 1.057 (MSE) to 0.794, a 19.6% reduction against EPL (95% CI of the difference [−0.257, −0.136], p < 0.001). nMAE_all rises from 0.620 to 0.785. The revised Table 1 caption states this trade-off explicitly.
W1 — Novelty: “a station-specific coefficient added to EPL.” We have restated the contribution. It is not the asymmetric weight itself, which many losses share, nor a new architecture, but a training procedure: the under-prediction rate, measured on a held-out period, adjusts the asymmetry of the training loss while the model trains, with one step size chosen on a three-value grid. SCAL is not tied to EPL. It multiplies the loss on extreme under-predictions by (1+κᵢ) and updates κᵢ from the under-rate on a held-out calibration period; κ is kept per station because the bias differs between stations, but we do not claim that this beats a shared κ (General Response). Two results support it as a method rather than a coefficient:
It acts on whichever base it is added to. On all six bases (MSE, ExLoss, EPL, LINEX, Expectile, Pinball) it brings the under-rate from 0.49–0.86 into 0.52–0.58. On the five bases that under-predict it moves the bias toward zero and raises POD, and on MSE, ExLoss, EPL and LINEX it also lowers nMAE_ext and raises SEDI (all p < 0.001). On Expectile it corrects the direction while the change in error is not statistically significant, and the tail error of Expectile 0.9 + SCAL is not significantly different from that of the tuned Expectile 0.95, at 0.092 lower nMAE_all. On Pinball, whose rate is already near 0.5, the changes are not statistically significant (General Response).
The feedback signal must come from held-out data. In the submitted pipeline, driven by training batches, κ collapsed as the model fit the training data and the effect was lost by the end of training. Driven by held-out calibration data, the effect persists to the last epoch (nMAE_ext 0.916 vs 1.117 for EPL).
We do not claim a theoretical contribution, and τ=0.5 is an interpretable reference rather than a derived optimum (sRue W2). On the architecture: we agree that factorized spatial and temporal attention is not new, and we do not claim the architecture as a contribution. The station-aware cross-attention and adapter are one design choice for reading the wide-area field out per station, and the directional correction persists without them (Q1, Q2).
W2 — Other methods and a single dataset. We added ExLoss (Xu et al., 2024) as a sixth base loss, trained with and without SCAL, validation-tuned asymmetric losses (Pinball, Expectile, LINEX), fixed multipliers, station-wise post-processing (bias correction, quantile mapping, extreme-range linear), a matched-cost comparison and relative economic value (sRue W1/W3, A3n1 Q3). The single network of 18 KMA stations and the single reanalysis input remain limitations, which we now state explicitly (Section 5, Limitations).
Q1 — Does SCAL transfer to other backbones? Yes. On a ConvLSTM backbone, EPL + SCAL (η=8) lowers nMAE_ext from 1.185 to 0.787, close to the Transformer’s 0.794, and the under-rate from 0.861 to 0.550, relative to EPL on the same backbone. Reaching this level costs more on the ConvLSTM: nMAE_all rises from 0.630 to 1.210 (0.620 to 0.785 on the Transformer), and CSI and SEDI are lower (0.207 and 0.606 vs 0.301 and 0.722). The directional correction thus comes from the loss, and the backbone sets its cost.