- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces M²Weather, a benchmark for jointly studying multi-station and multi-variable weather forecasting. It constructs quality-controlled datasets at France, Europe, and Global scales, unifies the evaluation of SSSV, SSMV, MSSV, and MSMV forecasting paradigms, and proposes a lightweight adapter to add missing station or variable interactions to frozen models. Experiments on 16 baselines show that jointly modeling station and variable relationships generally performs best, while adding missing interactions consistently improves forecasting performance, with cross-station information showing particularly strong benefits.
Critical strengths
-
The paper provides a unified evaluation framework across France, Europe, and Global scales, with relatively careful station quality control and consistent training/evaluation protocols, which makes comparisons across different forecasting paradigms more reliable.
-
The paper evaluates 16 representative models and uses lightweight interaction adapters on frozen backbones to isolate the effect of adding missing station or variable interactions. The consistent gains across models and datasets provide evidence for the value of joint station-variable information.
Critical weaknesses
-
The dataset and benchmark are carefully constructed, but compared with prior station-weather benchmarks such as Weather5K, the data itself does not introduce a fundamentally new task or data source. The main empirical findings, such as the importance of cross-station information and the benefits of jointly modeling station and variable interactions, are largely consistent with well-established understanding in weather forecasting. Although the paper provides empirical evidence, the amount of new scientific insight appears limited.
-
Although the paper discusses multiple MSMV approaches, only HiSTGNN is included as a native MSMV baseline in the unified benchmark. The authors explain that DIN and StationPDE could not be included due to unavailable implementations, but relying on a single MSMV model makes it difficult to support a strong paradigm-level conclusion that MSMV modeling is consistently superior. While this conclusion may be broadly consistent with common understanding in weather forecasting, it is also one of the paper’s central claimed insights, and the current experimental evidence is therefore insufficient to substantiate it convincingly.
Questions for authors
-
Since HiSTGNN is the only native MSMV baseline, how strongly do the authors believe the current experiments support a paradigm-level conclusion that MSMV modeling is superior? Is it possible to include additional MSMV baselines or provide stronger evidence beyond a single model?
-
What are the key capabilities of M²Weather that cannot already be studied with existing datasets such as Weather5K? In particular, what new scientific questions or findings are enabled by M²Weather beyond providing a more unified evaluation protocol?
-
Since the proposed adapter operates on frozen model outputs, to what extent do the authors view these results as evidence for end-to-end joint station-variable modeling, rather than simply evidence that additional station and variable information can improve post-hoc predictions?
4 comments
Respond to Q2/W1: What research capabilities and insights does M²Weather add beyond Weather5K?
Thank you for the review. We would like to clarify that M²Weather and Weather5K are based on different data products. Weather5K is based on NOAA's Integrated Surface Database (ISD), while M²Weather is based on the global land surface observation product released by C3S/CDS. Specifically, C3S/CDS has many advantages over NOAA ISD, as shown in the table below. NOAA ISD has accumulated more than 35,000 stations, with the earliest records dating back to 1901; while C3S/CDS includes 34,729 intraday stations, 86,499 daily stations, and 83,148 monthly stations, with the earliest records traceable back to 1718. NOAA ISD focuses on surface observations, including temperature, dew point, wind, pressure, precipitation, visibility, etc.; C3S/CDS additionally provides atmospheric variables at 37 pressure levels. NOAA ISD data is mainly provided as CSV files; C3S/CDS additionally provides standardized NetCDF files for convenient direct reading of variables.
| Station and the degree of spatiotemporal coverage | Number of observed variables | Ease of data acquisition | |
|---|---|---|---|
| NOAA ISD (Weather5K) | Middle | Middle | Middle |
| C3S/CDS (M²Weather) | High | High | High |
Furthermore, compared with existing benchmarks such as Weather5K, in addition to our systematic validation of the importance of joint station-variable modeling, our benchmark also provides two additional contributions and insights:
- We correct misleading comparisons in the literature. Specifically, channel interactions in multivariate time-series models are typically designed to model relationships among different variables. When adapting such models, existing benchmarks have treated different stations as channels and used the same mechanism to model spatial relationships among stations, thereby conflating the two types of interactions in their comparisons, such as Corrformer in weather5k. Cross-variable dependencies and cross-station spatial dependencies are distinct modeling targets. To address this issue, we use model wrappers to distinguish the two types of interactions and fairly compare multi-station single-variable (MSSV) and single-station multi-variable (SSMV) models under a unified forecasting task and evaluation protocol.
- We propose a multi-criteria quality control and station stratification strategy. Weather5K selects stations based on a threshold of more than 90% valid hourly records. We additionally examine observation coverage for each variable, consecutive gaps, quality control pass rates, and station metadata consistency. This allows us to identify stations with high overall coverage but prolonged gaps for individual variables, and to select stations separately for the global and regional benchmarks.
Respond to Q3: Do the adapter results support end-to-end joint modeling of station and variable interactions?
Thank you for the review. To demonstrate that our adapter can improve forecasts beyond post-processing, we selected four backbone models of different types on the France dataset and compared three settings: the original backbone, fitting the adapter separately after freezing the backbone, and adding the adapter from the start of training and optimizing it jointly with the backbone. For all four models, both settings with the adapter outperform the corresponding original backbone. This shows that the complementary station and variable information introduced by the adapter can improve predictions both by correcting the outputs of frozen models and through end-to-end training. These results provide further empirical evidence for our main conclusion that jointly using station and variable interactions improves station weather forecasting performance.
| Model | Original | + Adapter | Joint Training |
|---|---|---|---|
| DLinear | 0.4979 ± 0.0005 | 0.4605 ± 0.0000 | 0.4698 ± 0.0001 |
| S²Transformer | 0.3784 ± 0.0003 | 0.3657 ± 0.0001 | 0.3466 ± 0.0001 |
| TQNet | 0.3703 ± 0.0004 | 0.3483 ± 0.0002 | 0.3513 ± 0.0004 |
| HiSTGNN | 0.3089 ± 0.0044 | 0.3018 ± 0.0041 | 0.3011 ± 0.0056 |
Response to Authors
Thank you for the additional baseline, ablations, and joint-training experiments. These strengthen the evidence and address several of my concerns. However, my main concern about novelty and new scientific insight remains, consistent with Reviewer DzTY’s observation. Joint cross-station and cross-variable modeling has already been studied, and further confirming its benefits offers limited novelty. While I appreciate the improved quality control and unified evaluation protocol, I am not convinced that the benchmark’s incremental contribution meets the bar for ICLR. I therefore maintain my original assessment.
Respond to Q1/W2: Can additional MSMV baselines and HiSTGNN ablations strengthen the evidence for joint modeling?
We also report the result of the most recent MSMV method, StationPDE [1] (September, 2026) on the France dataset below. StationPDE still outperforms all SSSV, MSSV, and SSMV methods in the paper, which further confirms the importance of jointly modeling variable and station interactions.
To provide more convincing evidence, we also conducted ablation experiments on HiSTGNN using the France dataset by removing variable interactions and station interactions separately. The results show that performance consistently declines when either type of interaction is removed, further confirming our main conclusion. We will subsequently report StationPDE results on all datasets.
[1] Wang X, Chen C, Li R, et al. StationPDE: Station-Oriented Surface PDE Learning for Multi-Station Multivariate Weather Forecasting[J]. arXiv preprint arXiv:2609.22123, 2026.