acceptodds
Under review as a conference paper at ICLR 2027

SCAL: A Station-Calibrated Asymmetric Loss for Extreme-Aware Spatiotemporal Wind Forecasting

Abstract

Forecasting extreme wind speed is difficult: tail events are sparse, their errors carry a systematic direction, and each station behaves differently. We propose the Station-Calibrated Asymmetric Loss (SCAL), which sets the penalty in the extreme range, the upper 10% of each station’s training distribution, from that station’s own under-prediction rate during training, with hyperparameters shared rather than tuned per station. We combine it with a spatiotemporal Transformer that turns the preceding 24 h of the wide-area ERA5 field into per-station representations, forecasting the next 24 h at 18 stations operated by the Korea Meteorological Administration. Applied to the Self-adaptive Extreme Penalized Loss (EPL) over three random seeds on the 2023–2024 test period, SCAL moves the extreme-range bias from −0.905 to +0.056 m s⁻¹ and the under-prediction rate from 0.728 to 0.482, against a neutral value of 0.5. Across objectives trained under one backbone and schedule, the size of the extreme-range error and its direction prove separable: those whose asymmetry is fixed in advance lower the error yet still fall below the observation at 62% to 78% of extreme observations. Of those, only quantile regression at the 0.9 level brings the rate close to neutral, and at a higher whole-range cost than SCAL (0.817 against 0.681 in normalized mean absolute error). The extreme-range error falls 13.9% relative to EPL while the whole-range error rises 32.8%.

Primary area:
learning on time series and dynamical systems
Keywords:
wind forecasting, spatiotemporal forecasting, extreme value prediction, asymmetric loss, station calibration, ERA5 reanalysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m29SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issue of extreme wind speed prediction and proposes a loss function called "Station-Calibrated Asymmetric Loss" (SCAL), which is combined with the spatio-temporal Transformer method to construct a complete prediction framework. This framework converts the large-scale ERA5 meteorological field data from the past 24 hours into data representations specific to each site, thereby predicting the wind speed for 18 stations under the Korea Meteorological Administration's jurisdiction in the next 24 hours. In the experiments, a baseline model was constructed by changing only the loss function while keeping the main network and training settings unchanged, thereby verifying the effectiveness of the proposed SCAL.

Critical strengths
  1. The author evaluated the model using multiple comprehensive indicators, including nMAE-ext, bias_ext, underreporting rate (under rate), POD, SEDI, etc.
  2. This paper focuses on the "systematic bias" issue in extreme weather forecasting; this is a problem of practical significance.
Critical weaknesses
  1. The primary innovation emphasized by the authors—the design of a loss function called SCAL—essentially amounts to introducing a position-specific calibration coefficient into the "extreme underestimation" mechanism of the existing EPL method. Regarding the architecture, the authors employ a decoupled modeling approach, utilizing independent spatial and temporal self-attention mechanisms; however, given the extensive existing research and established findings in this field, this approach is overly simplistic. Furthermore, SCAL lacks both a theoretical foundation and sufficient empirical validation to demonstrate its effectiveness.
  2. The main experiments establish a baseline solely by modifying the loss function, without comparing the approach against other relevant methods. The scope of the experiments is limited, covering only a single dataset, which is insufficient to demonstrate the method's generalization capability and robustness. Moreover, Table 1 fails to show that SCAL performs well; on the contrary, it indicates that SCAL actually leads to an increase in both overall error and extreme error.
Questions for authors

Q1. Are there any relevant experiments or other evidence indicating that the SCAL loss function (as well as the loss functions used in other backbone networks) is not only applicable to the architecture proposed by the authors but also can bring benefits to it? Q2. The ablation experiments show that removing the "site-specific representation path" will cause a deviation of "under-prediction" in the regression prediction. Does this mean that if the backbone network cannot identify site-specificity at the feature level and only relies on the penalty terms in the loss function, the model will be unable to learn the physical mapping relationship necessary to correct the deviation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer m29S (1/2)

Thank you for the expert review. We address the reading of Table 1 first, then novelty, scope and your two questions. Note that we found and fixed a 9-hour misalignment between observations and ERA5 inputs and retrained every model; all numbers below come from the corrected pipeline, and the General Response lists what the correction changed.

W2 — “SCAL increases both overall and extreme error in Table 1.” SCAL lowers extreme-range error and raises whole-range error. This was true in the submission (nMAE_ext 0.837 for EPL vs 0.721 for SCAL) and holds after correction: nMAE_ext falls from 0.988 (EPL) and 1.057 (MSE) to 0.794, a 19.6% reduction against EPL (95% CI of the difference [−0.257, −0.136], p < 0.001). nMAE_all rises from 0.620 to 0.785. The revised Table 1 caption states this trade-off explicitly.

W1 — Novelty: “a station-specific coefficient added to EPL.” We have restated the contribution. It is not the asymmetric weight itself, which many losses share, nor a new architecture, but a training procedure: the under-prediction rate, measured on a held-out period, adjusts the asymmetry of the training loss while the model trains, with one step size chosen on a three-value grid. SCAL is not tied to EPL. It multiplies the loss on extreme under-predictions by (1+κᵢ) and updates κᵢ from the under-rate on a held-out calibration period; κ is kept per station because the bias differs between stations, but we do not claim that this beats a shared κ (General Response). Two results support it as a method rather than a coefficient:

It acts on whichever base it is added to. On all six bases (MSE, ExLoss, EPL, LINEX, Expectile, Pinball) it brings the under-rate from 0.49–0.86 into 0.52–0.58. On the five bases that under-predict it moves the bias toward zero and raises POD, and on MSE, ExLoss, EPL and LINEX it also lowers nMAE_ext and raises SEDI (all p < 0.001). On Expectile it corrects the direction while the change in error is not statistically significant, and the tail error of Expectile 0.9 + SCAL is not significantly different from that of the tuned Expectile 0.95, at 0.092 lower nMAE_all. On Pinball, whose rate is already near 0.5, the changes are not statistically significant (General Response).

The feedback signal must come from held-out data. In the submitted pipeline, driven by training batches, κ collapsed as the model fit the training data and the effect was lost by the end of training. Driven by held-out calibration data, the effect persists to the last epoch (nMAE_ext 0.916 vs 1.117 for EPL).

We do not claim a theoretical contribution, and τ=0.5 is an interpretable reference rather than a derived optimum (sRue W2). On the architecture: we agree that factorized spatial and temporal attention is not new, and we do not claim the architecture as a contribution. The station-aware cross-attention and adapter are one design choice for reading the wide-area field out per station, and the directional correction persists without them (Q1, Q2).

W2 — Other methods and a single dataset. We added ExLoss (Xu et al., 2024) as a sixth base loss, trained with and without SCAL, validation-tuned asymmetric losses (Pinball, Expectile, LINEX), fixed multipliers, station-wise post-processing (bias correction, quantile mapping, extreme-range linear), a matched-cost comparison and relative economic value (sRue W1/W3, A3n1 Q3). The single network of 18 KMA stations and the single reanalysis input remain limitations, which we now state explicitly (Section 5, Limitations).

Q1 — Does SCAL transfer to other backbones? Yes. On a ConvLSTM backbone, EPL + SCAL (η=8) lowers nMAE_ext from 1.185 to 0.787, close to the Transformer’s 0.794, and the under-rate from 0.861 to 0.550, relative to EPL on the same backbone. Reaching this level costs more on the ConvLSTM: nMAE_all rises from 0.630 to 1.210 (0.620 to 0.785 on the Transformer), and CSI and SEDI are lower (0.207 and 0.606 vs 0.301 and 0.722). The directional correction thus comes from the loss, and the backbone sets its cost.

AuthorsOpenReview ↗

Response to Reviewer m29S (2/2)

Q2 — Without station-specific features, can the loss alone correct the bias? Partly. Removing the station adapter, or the whole station representation path (station-aware cross-attention and adapter, after which the decoder query is identical for all stations), leaves the tail error and direction not significantly different from those of the full model (nMAE_ext 0.784 and 0.808 vs 0.794; under-rate 0.520 and 0.562 vs 0.541; with MSE + SCAL, 0.740 and 0.755 vs 0.734), and the ConvLSTM result (Q1), which has no station-specific path in the encoder, points the same way. The directional correction therefore does not require these two components, and we correct the submission’s statement that these components are required for it. However, the decoder still reads the encoder tokens at each station’s coordinates, so station location information remains available, and these variants are not an independent test without station information. We no longer claim a performance benefit for these components: at the selected checkpoints whole-range error differs between variants (0.884 and 0.778 vs 0.785), but variants checkpointed earlier, when κ is larger, have higher nMAE_all, and at the last epoch before early stopping all three lie within 0.68–0.69 on test. An attention analysis in the appendix shows that the cross-attention reads upstream areas (the northwestern continent and the southern sea) at up to about 2.7 times the uniform weight; we give this as a description of behavior, not as evidence that the component is needed.

Thank you for your thorough response. I have carefully read all of your comments, but we believe that several concerns remain unresolved: $ \textbf{1.}$ The paper does not introduce any innovative modules or methods, and merely exploring different loss functions is insufficient to establish methodological novelty. $ \textbf{2.}$ The experiments are conducted on only one small-scale dataset, without sufficient evidence of generalization across multiple datasets. $ \textbf{3.}$ The experimental evaluation lacks comparisons with state-of-the-art (SOTA) methods, further limiting the credibility of the paper's claims regarding its effectiveness.

Based on these concerns, I maintain my original evaluation and score.

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for taking the time to carefully review our responses and revised manuscript. We appreciate your thoughtful feedback and acknowledge that the concerns you raised regarding methodological novelty, generalization across datasets, and comparisons with state-of-the-art methods remain important limitations of our work.

Your comments have helped us better understand the limitations of the current study and identify directions for further improvement. Thank you again for your time and constructive feedback.

Reviewer_A3n1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Station-Calibrated Asymmetric Loss (SCAL) for forecasting station-level extreme wind speeds. SCAL augments an extreme-aware base loss with a station-specific coefficient on under-predicted extreme observations. Each coefficient is updated during training from an exponential moving average of the station’s under-prediction rate relative to a target of 0.5. The loss is combined with a spatiotemporal Transformer that encodes a wide-area ERA5 field and produces 24-hour forecasts for 18 KMA stations through station-aware representations. On a temporally held-out 2023–2024 test period, SCAL reduces extreme-range error and moves the conditional signed bias and under-prediction frequency toward neutrality. The paper also reports the associated degradation in whole-range error and false alarms, and studies global versus station-specific calibration, spatial context, architectural components, and a ConvLSTM backbone.

Critical strengths
  1. The empirical comparison is unusually controlled and transparent. Most loss functions are evaluated with the same backbone, temporal split, training schedule, checkpoint criterion, and three random seeds. The paper reports not only extreme conditional errors and directional bias, but also whole-range degradation, probability of detection, false alarms, frequency bias, CSI, and SEDI. This makes the actual operating-point trade-off visible.
  2. The paper provides evidence that the feedback mechanism is not specific to one architecture. Applying SCAL to ConvLSTM also moves the conditional bias and under-prediction rate toward the target while reducing extreme-range error. The station/global coefficient comparison and station-wise analyses further support the claim that heterogeneous station behavior matters.
Critical weaknesses
  1. It is unclear why an under-prediction rate of 0.5 on observed extreme cases should be treated as a calibrated or practically optimal target, especially when SCAL substantially increases false alarms and whole-range error while failing to improve CSI.

  2. The benefit of station-specific calibration is not fully convincing, since a global adaptive coefficient performs similarly and simple post-hoc station-wise calibration is not included as a baseline.

Questions for authors
  1. Why is 0.5 the appropriate target for the under-prediction rate conditioned on observed extreme cases?

  2. How does SCAL perform under other target rates or application-specific miss and false-alarm costs?

  3. Can the authors compare SCAL with simple station-wise post-hoc bias correction or quantile mapping?

  4. Is the improvement of station-specific coefficients over a single global coefficient statistically significant?

  5. Was the reported η=8\eta=8 configuration selected using validation data only?

1 comment
AuthorsOpenReview ↗

Response to Reviewer A3n1

Thank you for recognizing the controlled comparison and the ConvLSTM result. Both are kept. Note that we found and fixed a 9-hour misalignment between observations and ERA5 inputs and retrained every model; all numbers below come from the corrected pipeline, and the General Response lists what the correction changed.

W1 / Q1 — Why 0.5? We agree that 0.5 should not be presented as a calibrated or optimal target, and we revise the wording. We chose τ=0.5 as an interpretable reference for the direction of error on observed extremes, not as a universally optimal or cost-neutral operating point: at τ=0.5 a realized extreme is as likely to be under- as over-predicted. The conditional-bias literature (Murphy & Winkler, 1987; Seo et al., 2018) motivates this target rather than derives it, and we report the mean form separately as bias_ext. Costs are evaluated separately (Q2). The CSI difference relative to EPL is not statistically significant (0.301 vs 0.300, p = 0.94).

Q2 — Other targets and costs. We did not rerun other τ values. Instead we evaluate costs directly with relative economic value: EPL + SCAL is the best model for cost/loss ratios α = 0.14–0.22, Pinball for 0.02–0.06, Expectile for 0.08–0.12 and MSE above 0.24 (sRue W1). The target is also reached across base losses: SCAL moves the under-rate of six different bases from 0.49–0.86 into 0.52–0.58 (General Response).

W2 / Q3 — Simple station-wise post-hoc correction. We added bias correction, quantile mapping and an extreme-range linear correction, each fitted per station on the 2020 calibration data and applied to test.

CorrectionMSE base: nMAE_ext / nMAE_all / underEPL base: nMAE_ext / nMAE_all / under
None1.057 / 0.541 / 0.8640.988 / 0.620 / 0.782
Bias correction1.114 / 0.531 / 0.8841.164 / 0.579 / 0.855
Quantile mapping1.040 / 0.578 / 0.7561.101 / 0.624 / 0.784
Extreme-range linear1.094 / 0.539 / 0.9151.086 / 0.602 / 0.939
+ SCAL (η=8), in training0.734 / 0.803 / 0.5770.794 / 0.785 / 0.541

The post-processing methods evaluated here leave the tail under-predicted and mostly raise tail error, so they did not achieve comparable directional correction; this does not rule out stronger input-dependent or tail-targeted post-processing. The same holds on Pinball, Expectile and LINEX bases (nMAE_ext 1.06–1.16).

Q4 — Is station-wise κ significantly better than global κ? No. A single shared κ gives a mean tail error close to that of station-wise κ (nMAE_ext 0.783 vs 0.794), and we withdraw the claim that station-wise κ improves the mean. Station-wise κ gives a more even result across stations in all six base × seed runs (sd 0.104 vs 0.131; worst station 1.11 vs 1.19), but these differences are not statistically significant (p = 0.24 and 0.19). What is significant is the calibration loop itself relative to no calibration: sd 0.192 to 0.104, worst station 1.56 to 1.11, and mean |under-rate − 0.5| 0.28 to 0.09 (all p < 0.001). The learned κ ranges from 3.3 to 20.0 across stations and is correlated with each station’s under-rate under EPL (Spearman 0.56, p = 0.017) and with its bias (−0.75, p < 0.001), so it tracks where the base model under-predicts. Individual stations show why the means are close: station-wise κ has lower tail error at Namhae (κ = 20.0), where under-prediction persists across years (validation 1.066 vs 1.157; test 1.108 vs 1.191), and higher at Seongsan (κ = 3.3), where the 2020 calibration data under-predicted little but the test period much (0.817 vs 0.684). We give this as an illustration without a significance claim.

Q5 — Was the configuration selected on validation only? In the revision, yes. For each base loss, η is chosen by the validation |bias_ext| closest to zero (η=8 for MSE, ExLoss, EPL and Pinball, 4 for LINEX, 2 for Expectile), tuned baselines by the same rule, and the checkpoint by validation nMAE_ext. These rules and the checkpoint criterion were fixed before any model of this revision was scored on the test period; the ExLoss + SCAL pair was added afterwards under the same rules. In the submission, η=8 had been chosen with test results visible; we disclose this, and the validation rule now selects the same value for EPL. Presenting the base-versus-SCAL pairs as the main result was decided after the test results were seen, so we report every pair and every metric.

Reviewer_sRueOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCAL, an adaptive asymmetric loss for station-level wind forecasting. Each station receives an additional penalty for underpredicting observations above its training-period 90th percentile. The penalty is updated during training using that station’s underprediction frequency.

The authors combine SCAL with a spatiotemporal Transformer that uses the preceding 24 hours of ERA5 fields to forecast 24 hours ahead at 18 Korean stations. Experiments cover a chronological train/validation/test split, three random seeds, several alternative losses, and architectural and spatial-range ablations.

Relative to EPL, SCAL reduces normalized tail MAE by 13.9% and largely removes average tail underprediction. This comes with a 32.8% increase in whole-range normalized MAE and substantially more false alarms.

Critical strengths
  1. Simple and implementable method. SCAL adds little machinery and avoids a separate manual penalty search for every station.

  2. Controlled loss comparisons. Holding the backbone and training schedule fixed makes the effects of changing the objective easier to interpret.

Critical weaknesses
  1. The practical value of the chosen operating point remains unclear. Relative to EPL, SCAL-EPL raises whole-range nMAE from 0.513 to 0.681. Its frequency bias reaches 2.154, its false alarm ratio is 0.586, and CSI falls from 0.392 to 0.363. These are substantial costs. The method may be useful when misses are especially expensive, but that requires an explicit cost-based evaluation rather than treating directional neutrality as the preferred outcome.

  2. A 50% underprediction rate on observed extremes is a design target, not a general calibration criterion. Conditioning on the realized observation being above P90 naturally selects cases that point forecasts may undershoot. Reaching 0.5 on this subset does not establish probabilistic calibration or operational optimality. The paper acknowledges that this is a directional diagnostic, but still gives neutrality considerable weight when selecting and interpreting results.

  3. The baseline comparison does not isolate the benefit of adaptation. SCAL receives a search over step sizes and two base losses, while LINEX, quantile, and expectile objectives use single asymmetry settings. Appendix G acknowledges this. A matched search, a fixed underprediction multiplier, and a validation-fitted station-wise post-processing correction are important controls. Without them, the improvement could largely reflect choosing a stronger upward correction.

Questions for authors

How does SCAL compare with tuned fixed penalties and station-wise post-hoc corrections at the same false alarm rate or whole-range error?

Was η = 8 selected using validation results alone? Please make the selection rule explicit, especially since it is not the best setting for tail MAE.

Does station-wise calibration consistently outperform the shared coefficient under paired weather-event resampling?

How sensitive are the results to checkpoint selection, and does a stable late-training version retain the reported benefits?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sRue (1/2)

Thank you for the precise review. Your points on the operating point, the role of τ, and isolating adaptation led directly to the new controls below. Note that we found and fixed a 9-hour misalignment between observations and ERA5 inputs and retrained every model; all numbers below come from the corrected pipeline, and the General Response lists what the correction changed.

W1 / Q1 — Practical value of the operating point, and comparison at equal cost.

Equal whole-range error. We added a station-wise offset to each baseline, fitted on validation so that its nMAE_all equals SCAL’s. At this matched cost, the differences in nMAE_ext and nMAE_all are not statistically significant (EPL base: ΔnMAE_ext −0.026). SCAL still has a lower under-rate (−0.126) and a bias closer to zero (+0.59), both p < 0.001. With the MSE base the under-rate is also lower (−0.069, p = 0.009), but CSI is lower by 0.042 (p < 0.001). At equal cost, an offset leaves forecasts biased low on extremes, while SCAL brings them closer to neutral.

Explicit costs. Relative economic value (Richardson, 2000) shows which model is best at each cost/loss ratio α: Pinball for α = 0.02–0.06, Expectile for 0.08–0.12, EPL + SCAL for 0.14–0.22, and MSE above 0.24. Validation shows the same pattern.

Detection scores. With the corrected pipeline, the CSI difference relative to EPL is not statistically significant (0.301 vs 0.300, p = 0.94). The higher FAR (0.52 to 0.65) is offset by higher POD (0.46 to 0.72).

We now present SCAL as most useful at intermediate miss costs, not as a uniform improvement, and state the increase in whole-range error and FAR in the Limitations. Because the inputs are ERA5 reanalysis, we also state there that use in warnings is a potential application rather than one demonstrated here.

W2 — τ=0.5 is a design target. We agree. Conditioning on observed extremes selects cases that point forecasts tend to undershoot, so a rate of 0.5 does not imply probabilistic calibration or operational optimality. We chose τ=0.5 as an interpretable reference for the direction of error on observed extremes, not as a universally optimal or cost-neutral operating point. It sets the median error on observed extremes to zero. The conditional-bias literature motivates it rather than derives it: the bias given the observation, E[f | x] − x, arises from the likelihood–base rate factorization (Murphy & Winkler, 1987), and Seo et al. (2018) call it type-II conditional bias, show that least-squares estimators are subject to it, and find that penalizing it lowers RMSE in the tails by 10–20% at a cost of 3% or less in overall RMSE, a trade-off of the same kind we report. Because that bias is a mean and our target a median rate, we also report the mean form, bias_ext. τ is set by design rather than selected on a score: it encodes no preference between missing and over-forecasting an observed extreme, and an application that weighs the two differently would set τ from that cost ratio. How the value of the forecasts varies with the cost ratio is reported through relative economic value (W1).

AuthorsOpenReview ↗

Response to Reviewer sRue (2/2)

W3 — Isolating the benefit of adaptation. We added the three controls you requested, each selected on validation.

ModelnMAE_extnMAE_allbias_extunder-rate
EPL + SCAL (η=8, mean κ ≈ 10.7 at the checkpoint)0.7940.785−0.1500.541
EPL, fixed κ=100.7870.833−0.3670.575
EPL, fixed κ=50.8140.837−0.4310.575
Pinball (q=0.9, tuned)0.6921.078−0.0650.486
Expectile (0.95, tuned)0.6670.966−0.2360.550
LINEX (a=2, tuned)0.7281.012−0.3040.566
EPL + station-wise quantile mapping1.1010.624−1.4170.784

Fixed multiplier. A fixed multiplier selected from {2, 5, 10} by the same validation rule takes κ=10, yet still under-predicts on validation (bias −0.16 vs +0.07 for SCAL). On test its tail and whole-range errors are not significantly different from SCAL’s, but SCAL’s bias is closer to zero (+0.22, p < 0.001) and its SEDI is higher (+0.025, p = 0.024). Fixed multipliers are also sensitive to the chosen value: with κ=2 the bias is lower by 0.71 and the under-rate higher by 0.13 (both p < 0.001). The feedback therefore does not improve accuracy over a well-chosen fixed multiplier; it sets the coefficient without a search over κ values, with a bias closer to zero, and only the global step size η is chosen, on a three-value grid.

Tuned asymmetric losses. Within the same loss family, adding SCAL reaches the tuned level at a lower whole-range cost: the tail error of Expectile 0.9 + SCAL is not significantly different from that of tuned Expectile 0.95 (0.684 vs 0.667, p = 0.49; under-rate 0.548 vs 0.550), with 0.092 lower nMAE_all (p < 0.001) and 0.011 higher CSI (p = 0.002), and the same holds for LINEX 1 + SCAL against tuned LINEX 2 (0.742 vs 0.728, p = 0.63; 0.219 lower nMAE_all, p < 0.001). Across families, we state plainly that tuned Pinball, Expectile and LINEX reach lower nMAE_ext than EPL + SCAL (by 0.07–0.13), while EPL + SCAL has lower nMAE_all (by 0.18–0.29) and similar under-rates.

Station-wise post-processing (bias correction, quantile mapping, extreme-range linear correction, fitted on the 2020 calibration data) does not remove tail under-prediction and mostly raises nMAE_ext (1.04–1.16 across bases). The post-processing methods evaluated here did not achieve comparable directional correction; this does not rule out stronger input-dependent or tail-targeted post-processing.

Q2 — Was η=8 selected on validation only? In the submission it was not, and we thank you for asking. The revision applies an explicit rule: the η whose validation |bias_ext| is closest to zero. For EPL this rule selects η=8 (validation bias +0.07, vs −0.12 for η=4 and −0.26 for η=2), and η=8 also has the lowest test nMAE_ext (0.794, vs 0.801 and 0.815; observed after selection). Applied to each base, the rule selects η=8 for MSE, ExLoss and Pinball, η=4 for LINEX and η=2 for Expectile. On MSE and ExLoss the validation bias is still negative at η=8, the top of the grid (−0.17 and −0.10), so their reported correction is conservative. As a check made after the test results were available, selecting η by validation nMAE_ext instead changes the test differences by at most 0.009. All η values for every base are reported in the appendix (Tables 4 and 5).

Q3 — Station-wise vs shared κ under paired resampling. A single shared κ gives a mean tail error close to that of station-wise κ (nMAE_ext 0.783 vs 0.794), and we withdraw the claim that station-wise κ improves the mean. Station-wise κ gives a more even result across stations in all six base × seed runs (sd of nMAE_ext 0.104 vs 0.131; worst station 1.11 vs 1.19), but these differences are not statistically significant (p = 0.24 and 0.19). The significant effect is that of the calibration loop itself relative to EPL (sd 0.192 to 0.104, worst station 1.56 to 1.11, mean |under-rate − 0.5| 0.28 to 0.09, all p < 0.001).

Q4 — Checkpoint sensitivity. You identified a real problem. With κ driven by training batches, the benefit existed only early in training. With κ driven by the held-out calibration data, the benefit persists. At the last epoch before early stopping, EPL + SCAL still has lower nMAE_ext and under-rate than EPL, by 0.20 and 0.23 (0.916 vs 1.117; 0.579 vs 0.810). Best epochs are 14, 8 and 19, and early stopping ends training at epochs 34, 28 and 39.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for careful and constructive reviews. Before answering individually, we report one correction, what it changed, and how we now present the work.

Data-alignment error, found and fixed. While re-auditing the pipeline we found that KMA observations (timestamped in KST) were joined to ERA5 fields (UTC) by label, so the targets were shifted by 9 hours. The submitted “24-hour forecast” therefore covered 8 h before to 15 h after the last input time, so part of each target window overlapped the input period. We corrected the alignment, fixed a rounding error in the target-time embedding (the hour was truncated rather than rounded), and retrained every model. All numbers below and in the revised PDF come from the corrected pipeline, and the same corrected data, splits and training protocol apply to every baseline.

What the correction changed. Every target now lies 1–24 h after the input. Because the models were retrained together with the other changes listed below (one training protocol for all losses, held-out calibration data for κ), the differences cannot be attributed to the alignment alone. (a) Errors are higher for every model: EPL nMAE_ext 0.837 to 0.988 and nMAE_all 0.513 to 0.620. (b) The direction of the main effect of SCAL on EPL is the same: under-rate 0.728 to 0.482 in the submission and 0.782 to 0.541 now, and nMAE_ext −13.9% in the submission and −19.6% now, at a whole-range cost in both. (c) Two claims of the submission are no longer supported, and we withdraw them below: that station-wise κ gives a lower mean error than a shared κ, and that the station-aware cross-attention and station adapter are needed for the correction.

How we now present the work. Following (c), we no longer present the paper as a new Transformer architecture or as evidence that station-wise calibration is better. We present SCAL as a training procedure that adjusts the asymmetry of a base loss while the model trains, with a conceptually simple rule: three hyperparameters shared by all stations, with no per-station search, and one step size chosen on a three-value grid. SCAL is one way to adjust the direction of error on observed extremes; its effect and cost depend on the base loss and on the operating point it is compared with. The Transformer is the testbed. Because the inputs are ERA5 reanalysis, which is produced after the fact, we describe use in warnings or disaster response only as a potential application.

Changes made in response to the shared concerns.

(1) One fixed training protocol for all losses. The submitted hyperparameters had been tuned for a different loss. They are replaced by common settings (d_model 64, dropout 0.1, weight decay 0.01, AdamW 3e-4 with warmup and cosine decay), identical for every loss. The checkpoint is the epoch with minimum validation nMAE_ext after warmup, and training stops early on the same metric with patience 20.

(2) Validation-only selection. For each base loss, η is the value in {2, 4, 8} whose validation |bias_ext| (seed mean) is closest to zero: η=8 for MSE, ExLoss, EPL and Pinball, η=4 for LINEX and η=2 for Expectile. Tuned baseline levels use the same rule. The η grid, the selection rules and the checkpoint criterion were fixed before any model of this revision was scored on the test period; the ExLoss + SCAL pair was added afterwards under the same rules.

(3) κ is driven by held-out calibration data. The submission updated κ from the training-batch under-rate. That rate reaches τ as the model fits the training data, so κ collapsed late in training (sRue Q4). SCAL now updates κ from the under-rate on 2020, which is excluded from gradient updates. Every model, baselines included, is trained on 2014–2019. κᵢ starts at zero, so training begins from the base loss; the moving average of the rate starts from the first epoch’s value; and κᵢ is clipped to [0, κmax] with κmax = 100. The station threshold qᵢ (P90) and scale σᵢ are computed on 2014–2020 observations (training and calibration data).

(4) The role of τ=0.5. We chose τ=0.5 as an interpretable reference for the direction of error on observed extremes, not as a universally optimal or cost-neutral operating point. At τ=0.5 a realized extreme is as likely to be under- as over-predicted; a rate far from 0.5 means a systematic one-sided error on extremes, and 0.5 prefers neither error. We cite the conditional-bias literature (Murphy & Winkler, 1987; Seo et al., 2018) for why point forecasts under-predict observed extremes, not as a derivation of τ, and report the mean form, bias_ext, separately. We keep the method’s name, in which “calibrated” refers to setting each station’s κᵢ from held-out data, but we no longer describe the resulting forecasts as calibrated, and costs are evaluated explicitly (relative economic value, sRue W1).

AuthorsOpenReview ↗

General Response (2/3)

Main result: SCAL added to six base losses (test 2023–2024, mean of 3 seeds). SCAL adjusts the asymmetry of a base loss, so we evaluate it on several bases: each base is trained alone and with SCAL, with the same backbone and protocol and with η selected per base on validation. The aim is to reduce under-prediction of extremes. The upper block lists the six base losses trained alone, and the lower block the same bases with SCAL added. Bold marks a significant improvement over the same base loss (p < 0.05) under a paired moving-block bootstrap (5-day blocks; for each seed the two models are resampled jointly 1000 times, and the 95% interval and p are taken from the replicates pooled over the three seeds; every difference at p < 0.001 has the same sign in all three seeds, and p values are unadjusted).

LossnMAE_extnMAE_allunder-ratebias_extPODSEDICSI
Base losses
MSE1.0570.5410.864−1.740.3840.6070.301
ExLoss1.0030.5580.817−1.530.4370.6260.316
EPL0.9880.6200.782−1.360.4590.6130.300
LINEX (a = 1)0.8350.7320.730−1.010.5680.6490.295
Expectile (0.9)0.7140.8780.639−0.640.6890.7040.293
Pinball (0.9)0.6921.0780.486−0.070.8110.7260.263
Base loss + SCAL
MSE + SCAL (η = 8)0.7340.8030.577−0.390.7200.7190.296
ExLoss + SCAL (η = 8)0.7200.8360.561−0.320.7380.7220.289
EPL + SCAL (η = 8)0.7940.7850.541−0.150.7150.7220.301
LINEX (a = 1) + SCAL (η = 4)0.7420.7940.567−0.310.7200.7280.305
Expectile (0.9) + SCAL (η = 2)0.6840.8740.548−0.230.7530.7400.301
Pinball (0.9) + SCAL (η = 8)0.6860.9440.522−0.170.7780.7460.293

Three patterns hold across the six bases. First, direction: SCAL brings every under-rate into 0.52–0.58 (from 0.49–0.86), and on the five bases that under-predict it moves bias_ext toward zero and raises POD (all p < 0.001). Second, tail error: on MSE, ExLoss, EPL and LINEX, nMAE_ext falls by 0.32, 0.28, 0.19 and 0.09 and SEDI rises (all p < 0.001). On Expectile, SCAL corrects the direction and raises POD by 0.064 (p < 0.001); the changes in nMAE_ext (0.714 to 0.684, p = 0.17), nMAE_all (p = 0.67), FAR, CSI and SEDI are not statistically significant. On Pinball, whose rate is already near 0.5, no difference is statistically significant (nMAE_ext −0.007, p = 0.56; POD −0.033, p = 0.66), so we do not claim that SCAL improves every base. Third, the cost depends on the base. Where the base under-predicts strongly, the tail gain comes with higher whole-range error (+0.26, +0.28, +0.17 and +0.06 on MSE, ExLoss, EPL and LINEX, p < 0.001) and higher FAR, while the change in CSI is not statistically significant. Within one loss family, adding SCAL comes close to the tuned level at a lower cost: the tail error of Expectile 0.9 + SCAL is not significantly different from that of the validation-tuned Expectile 0.95 (0.684 vs 0.667, p = 0.49), with 0.092 lower nMAE_all (p < 0.001) and 0.011 higher CSI (p = 0.002), and the same holds for LINEX 1 + SCAL against tuned LINEX 2 (0.742 vs 0.728, p = 0.63; 0.219 lower nMAE_all, p < 0.001). SCAL therefore acts on the base it is added to, rather than as one more competing loss. Presenting these base-versus-SCAL pairs as the main result was decided after the test results were seen, so we report every pair and every metric.

For EPL, the base used in the submission, the paired bootstrap gives ΔnMAE_ext −0.194 (19.6%; 95% CI [−0.257, −0.136], p < 0.001), ΔSEDI +0.109 ([+0.081, +0.136], p < 0.001) and ΔnMAE_all +0.165 (26.6%); the change in CSI is not statistically significant (+0.001, p = 0.94), as POD rises from 0.46 to 0.72 and FAR from 0.52 to 0.65. For MSE, ΔnMAE_ext is −0.324 (95% CI [−0.367, −0.284], p < 0.001). For ExLoss (Xu et al., 2024), rerun on the corrected pipeline, ΔnMAE_ext is −0.284 (95% CI [−0.353, −0.236], p < 0.001).

Claims we revise.

AuthorsOpenReview ↗

General Response (3/3)

Station-wise vs shared κ: a single shared κ gives a mean tail error close to that of station-wise κ (nMAE_ext 0.783 vs 0.794; under-rate 0.525 vs 0.541), and we withdraw the claim that station-wise κ improves the mean. Station-wise κ gives a more even result across stations in all six base × seed runs (sd of nMAE_ext 0.104 vs 0.131; worst station 1.11 vs 1.19), but these differences are not statistically significant (p = 0.24 and 0.19). The significant station-level effect is that of the calibration loop itself: relative to EPL, SCAL lowers the cross-station sd of nMAE_ext from 0.192 to 0.104, the worst station from 1.56 to 1.11, and the mean station |under-rate − 0.5| from 0.28 to 0.09 (all p < 0.001).

Cross-attention and station adapter: the directional correction does not require them. Without the adapter, or without both the cross-attention and the adapter (the whole station representation path), the tail error and under-rate of SCAL are not significantly different from those of the full model (nMAE_ext 0.784 and 0.808 vs 0.794; under-rate 0.520 and 0.562 vs 0.541; m29S Q2). The same holds with MSE + SCAL, run in the revision (0.740 and 0.755 vs 0.734; under-rate 0.544 and 0.584 vs 0.577). We withdraw the submission’s claim that these components are needed for the correction and no longer claim a performance benefit for them: at the selected checkpoints whole-range error differs between variants (0.884 and 0.778 vs 0.785), but the variants stop at different epochs with different κ, and at the last epoch before early stopping all three lie within 0.68–0.69 on test. The decoder still reads the encoder at the station coordinates in these variants, so they are not a test without station information.

Input radius: with EPL + SCAL, shrinking the input has no statistically significant effect on tail error (0.773–0.794). Restricting the input to station neighborhoods (r = 2 and 5, at most 10% of the grid) raises test nMAE_all to 1.028 and 0.969 vs 0.785 (p ≤ 0.001), and the rise is smaller at r = 20 (0.908, p < 0.001). CSI and SEDI also decline as the input narrows (0.301 and 0.722 on the full domain, 0.231 and 0.636 at r = 2). With MSE + SCAL the tail error is likewise unchanged (0.732–0.768 vs 0.734), while the narrowest input raises nMAE_all (0.855 vs 0.803) and lowers CSI and SEDI. The tail correction comes from the loss, and the wide input lowers its whole-range cost and helps event detection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.