- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the different evolution rates of fast and slow variables in subseasonal-to-seasonal forecasting. It extends ESFM with a Temporal Attention Aggregator (TAA), mixed-lead-time training and inference, residual prediction for slow variables, and block rollout finetuning. The reported results show improved forecasting performance for several slow variables and competitive results against existing models on selected variables.
Critical strengths
-
The method combines temporal attention aggregation with mixed lead times to accommodate the different evolution rates of fast and slow variables within one model, while retaining frequent forecast outputs. This is a practically useful direction.
-
The experiments show improvements for several slow variables and evaluate the model using pointwise errors, anomaly correlations, and spectral metrics, providing supporting evidence for the approach.
Critical weaknesses
-
The final model combines TAA and mixed lead times with residual prediction, latent-channel reassignment, and block rollout finetuning. The paper examines several of these choices, but it is still hard to tell how much of the improvement comes from TAA and mixed-LT themselves. Comparisons with the same input history, initialization, and training budget would make their contributions clearer. In particular, comparing TAA with a simpler way of combining the same past inputs would help show whether the learned attention is needed.
-
The metrics and spectral analysis are useful, but there are few direct comparisons of the predicted fields. The appendix shows some failure cases, while the drought examples mainly track regional averages, which can hide differences in where and how anomalies develop. Maps comparing the final model, baselines, and ERA5 at several lead times would make these differences easier to assess. Showing slow variables alongside related atmospheric fields would also help check whether their evolution is consistent, beyond improvements in their individual scores.
Questions for authors
-
Under matched inputs, initialization, and training budgets, how much does each of TAA and mixed lead times contribute? Can targeted ablations verify their independent effects?
-
Can the authors compare forecast fields, reference fields, and spatial errors for representative fast and slow variables at different lead times, to show whether the metric improvements correspond to more realistic spatial structures and temporal evolution?
4 comments
Thank you for the response and additional experiments. Several of my concerns have been addressed. Two questions remain:
- Do the ablations in Appendix M use identical historical inputs and include the final block-rollout fine-tuning?
- T2m performs substantially worse than climatology. Could you explain this gap and compare with FuXi-S2S under the same evaluation protocol to clarify the model’s S2S advantage?
We are happy that our answers helped to clarify some ambiguities.
Ad 1. All ablations in Appendix M use identical inputs, but none of them include block rollout finetuning. The latter is not added, as it introduces substantial computational cost to each training run and may confound the ablation under study. All of the models have been compared under fair conditions. In particular, we note that the base mixed-LT 6/48h was also not rollout-finetuned.
Ad 2. As you accurately noticed, 2m-temperature performs worse than climatology on S2S timescales in RMSE metric but remains below the theoretical threshold of . As RMSE measures point-wise accuracy, it rewards oversmoothed predictions that minimize single-pixel errors. Climatology maps are smoother than those created by TAAM-S2S, meaning that our model is penalized for creating more realistic predictions. To support this claim, we attach the mean LSE (Logarithmic Spectral Error) over 48 test-time predictions for the 15th and 30th day of rollout for TAAM-S2S compared with Climatology. Furthermore, we break down the score to report LSE for small- and large-scale features separately (split at , which corresponds to east-west and north-south), showing that TAAM-S2S captures the variance of both significantly better than climatology.
| lead time (features) | +15 days (all) | +15 days (large-scale) | +15 days (small-scale) | +30 days (all) | +30 days (large-scale) | +30 days (small-scale) |
|---|---|---|---|---|---|---|
| TAAM-S2S | 0.144 ± 0.020 | 0.186 ± 0.034 | 0.115 ± 0.022 | 0.146 ± 0.027 | 0.183 ± 0.052 | 0.119 ± 0.021 |
| Climatology | 0.266 ± 0.054 | 0.311 ± 0.075 | 0.238 ± 0.050 | 0.268 ± 0.053 | 0.313 ± 0.071 | 0.239 ± 0.051 |
We ran a comparison with FuXi-S2S on 12 separate rollouts, but it must be noted that both models work on slightly different data. FuXi-S2S inputs and outputs are daily averages, whereas our model processes instantaneous values. To create daily means using TAAM-S2S, for each forecast, we started four rollouts at 00, 06, 12, 18 UTC on the initial date. Then we averaged those predictions to obtain mean values for each pixel. We used the same protocol to obtain the ground truth we are comparing with. This incurs a slight discrepancy with FuXi-S2S as its outputs are daily averages over 24 hours. However, we do not expect it to revert our conclusions.
| Metric | Model | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 |
|---|---|---|---|---|---|---|---|
| RMSE [K] | TAAM-S2S | 1.574 ± 0.158 | 2.590 ± 0.406 | 2.794 ± 0.348 | 2.877 ± 0.259 | 2.851 ± 0.318 | 2.810 ± 0.423 |
| RMSE [K] | FuXi-S2S | 1.360 ± 0.104 | 2.715 ± 0.357 | 3.191 ± 0.311 | 3.352 ± 0.357 | 3.509 ± 0.560 | 3.480 ± 0.636 |
| ACC | TAAM-S2S | 0.753 ± 0.032 | 0.318 ± 0.126 | 0.178 ± 0.113 | 0.124 ± 0.067 | 0.109 ± 0.114 | 0.106 ± 0.155 |
| ACC | FuXi-S2S | 0.833 ± 0.023 | 0.395 ± 0.088 | 0.189 ± 0.091 | 0.134 ± 0.101 | 0.095 ± 0.130 | 0.112 ± 0.124 |
| LSE | TAAM-S2S | 0.118 ± 0.008 | 0.171 ± 0.014 | 0.184 ± 0.022 | 0.189 ± 0.024 | 0.182 ± 0.019 | 0.178 ± 0.016 |
| LSE | FuXi-S2S | 0.250 ± 0.013 | 0.257 ± 0.022 | 0.273 ± 0.018 | 0.289 ± 0.016 | 0.307 ± 0.034 | 0.315 ± 0.027 |
We can see that TAAM-S2S achieves a lower RMSE than FuXi-S2S from week 2 onwards and keeps significantly better LSE throughout the whole predicted period. On the other hand, FuXi-S2S achieves slightly better ACC, although, with the exception of week 1, the scores are within one standard deviation.
With the discussion period closing soon, are there any remaining concerns that we could address?
Thank you kindly for the insightful review.
Adding ablation studies
To address your first concern and question, we added a section with ablation studies to the appendix. We compared the base mixed-LT 6/48h model with its versions without: TAA, slow variables latent level, residual prediction of slow variables, and the mixed-LT approach. All of the models are initialized from the same 48h baseline model and trained for the same number of epochs. In this particular case of S2S predictions of the mixed-LT 6/48h model, we observe that the majority of gain comes from the mixed-LT approach, as well as from changing the target of slow variables from absolute values to residuals. As reported in our response to reviewer 6XoP, since TAA does not significantly increase the number of parameters, one can reasonably expect that the processor compensates for any lack of representation with variable-specific patch embedding. However, it is important to note that TAA is promising to offer more flexibility, never degrades the metrics, and improves them for some variables (e.g., LSE of 6-hourly total precipitation).
In addition, TAA is more robust to perturbations in the inputs than the baseline variable-specific patch embedding. This robustness was quantified in the mixed-LT model described in Appendix G and trained with five lead times and the corresponding six past inputs (t_0, t_0 - 6h, t_0-48h, t_0-96h, t_0-192h, t_0-384h). The equivalent model with variable-specific patch embedding was trained under the same configuration. At inference time with delta_1=6h and delta_2=48h, inputs t_0-48h, t_0-96h and t_0-384h were replaced with Gaussian noise, leaving only t_0, t_0-6h, and t_0-192h as informative inputs. On fast variables, RMSE after two weeks increases by 10 to 80% with perturbed inputs and the variable-specific patch embedding, while our TAA module shows no noticeable degradation of the error. On slow variables, variable-specific patch embedding suffers from an error increase of 50 to 900%, while our TAA module limits the error increase to 7 to 64%. Sea surface temperature is the exception, as it is the most sensitive variable with TAA, increasing by up to 330%, which is nonetheless still lower than the 680% increase of variable-specific patch embedding. Therefore, although TAA shows moderate improvement over a linear combination of inputs in the baseline variable-specific patch embedding in the setup matching the training conditions, TAA is significantly more robust to perturbed inputs.
Qualitative results
Following your suggestions, in the revised version of the article, we also provide an additional section in the appendix, where we present predictions and error maps of baselines and the TAAM-S2S model. To cover as many variables as possible, we gave examples of mean sea-level pressure and 6-hourly total precipitation (fast surface variables), V-component of wind at 1000hPa and geopotential at 850hPa (atmospheric variables), and sea surface temperature and volumetric soil water layer 1 (slow variables). As challenging as it is to draw conclusions from prediction maps corresponding to one rollout, the figures show that TAAM-S2S doesn’t lose performance of 48h baseline on slow variables, and that they both supersede the 6h baseline under this regime.