acceptodds
Under review as a conference paper at ICLR 2027

WEATHER: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting

Abstract

Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather variables. However, existing studies often consider these relationships separately and use different datasets and experimental settings, hindering systematic assessment of their individual and joint contributions. In this paper, we introduce \benchmark, a benchmark for joint multi-station and multi-variable weather forecasting. Through multi-criteria quality control and station stratification, we collect 2,809 high-quality stations with 5 physically coupled weather variables across three spatial scales: France, Europe, and Global. This multi-scale design lets us examine whether conclusions persist from national to global station networks. We also introduce unified training and evaluation protocols to enable fair comparison of different station–variable modeling paradigms. To further examine the benefits of modeling station-variable relationships, we design a lightweight, plug-and-play adapter. With a trained weather forecasting model, this adapter can introduce missing station or variable relationships without retraining the model. This enables fair and efficient investigation of station–variable relationships. Systematic evaluation of 16 representative models shows the benefits of jointly modeling station and variable relationships. Completing missing relationships further reduces MSE for all adapted models on all three datasets. Together, these results identify the complementary information across stations and variables as an important resource for improving station weather forecasting.

Primary area:
datasets and benchmarks
Keywords:
Station Weather Forecasting, Multi-Station Multi-Variable Forecasting, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_abQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces M²Weather, a benchmark for jointly studying multi-station and multi-variable weather forecasting. It constructs quality-controlled datasets at France, Europe, and Global scales, unifies the evaluation of SSSV, SSMV, MSSV, and MSMV forecasting paradigms, and proposes a lightweight adapter to add missing station or variable interactions to frozen models. Experiments on 16 baselines show that jointly modeling station and variable relationships generally performs best, while adding missing interactions consistently improves forecasting performance, with cross-station information showing particularly strong benefits.

Critical strengths
  1. The paper provides a unified evaluation framework across France, Europe, and Global scales, with relatively careful station quality control and consistent training/evaluation protocols, which makes comparisons across different forecasting paradigms more reliable.

  2. The paper evaluates 16 representative models and uses lightweight interaction adapters on frozen backbones to isolate the effect of adding missing station or variable interactions. The consistent gains across models and datasets provide evidence for the value of joint station-variable information.

Critical weaknesses
  1. The dataset and benchmark are carefully constructed, but compared with prior station-weather benchmarks such as Weather5K, the data itself does not introduce a fundamentally new task or data source. The main empirical findings, such as the importance of cross-station information and the benefits of jointly modeling station and variable interactions, are largely consistent with well-established understanding in weather forecasting. Although the paper provides empirical evidence, the amount of new scientific insight appears limited.

  2. Although the paper discusses multiple MSMV approaches, only HiSTGNN is included as a native MSMV baseline in the unified benchmark. The authors explain that DIN and StationPDE could not be included due to unavailable implementations, but relying on a single MSMV model makes it difficult to support a strong paradigm-level conclusion that MSMV modeling is consistently superior. While this conclusion may be broadly consistent with common understanding in weather forecasting, it is also one of the paper’s central claimed insights, and the current experimental evidence is therefore insufficient to substantiate it convincingly.

Questions for authors
  1. Since HiSTGNN is the only native MSMV baseline, how strongly do the authors believe the current experiments support a paradigm-level conclusion that MSMV modeling is superior? Is it possible to include additional MSMV baselines or provide stronger evidence beyond a single model?

  2. What are the key capabilities of M²Weather that cannot already be studied with existing datasets such as Weather5K? In particular, what new scientific questions or findings are enabled by M²Weather beyond providing a more unified evaluation protocol?

  3. Since the proposed adapter operates on frozen model outputs, to what extent do the authors view these results as evidence for end-to-end joint station-variable modeling, rather than simply evidence that additional station and variable information can improve post-hoc predictions?

4 comments
AuthorsOpenReview ↗

Respond to Q1/W2: Can additional MSMV baselines and HiSTGNN ablations strengthen the evidence for joint modeling?

We also report the result of the most recent MSMV method, StationPDE [1] (September, 2026) on the France dataset below. StationPDE still outperforms all SSSV, MSSV, and SSMV methods in the paper, which further confirms the importance of jointly modeling variable and station interactions.

ModelMSEMAE
MSMV / StationPDE0.3062 ± 0.00140.3796 ± 0.0018

To provide more convincing evidence, we also conducted ablation experiments on HiSTGNN using the France dataset by removing variable interactions and station interactions separately. The results show that performance consistently declines when either type of interaction is removed, further confirming our main conclusion. We will subsequently report StationPDE results on all datasets.

HiSTGNN settingMSEMAE
Without variable interactions0.3098 ± 0.00480.3841 ± 0.0011
Without station interactions0.3150 ± 0.00010.3887± 0.0004
Full model0.3089 ± 0.00440.3736 ± 0.0014

[1] Wang X, Chen C, Li R, et al. StationPDE: Station-Oriented Surface PDE Learning for Multi-Station Multivariate Weather Forecasting[J]. arXiv preprint arXiv:2609.22123, 2026.

AuthorsOpenReview ↗

Respond to Q2/W1: What research capabilities and insights does M²Weather add beyond Weather5K?

Thank you for the review. We would like to clarify that M²Weather and Weather5K are based on different data products. Weather5K is based on NOAA's Integrated Surface Database (ISD), while M²Weather is based on the global land surface observation product released by C3S/CDS. Specifically, C3S/CDS has many advantages over NOAA ISD, as shown in the table below. NOAA ISD has accumulated more than 35,000 stations, with the earliest records dating back to 1901; while C3S/CDS includes 34,729 intraday stations, 86,499 daily stations, and 83,148 monthly stations, with the earliest records traceable back to 1718. NOAA ISD focuses on surface observations, including temperature, dew point, wind, pressure, precipitation, visibility, etc.; C3S/CDS additionally provides atmospheric variables at 37 pressure levels. NOAA ISD data is mainly provided as CSV files; C3S/CDS additionally provides standardized NetCDF files for convenient direct reading of variables.

Station and the degree of spatiotemporal coverageNumber of observed variablesEase of data acquisition
NOAA ISD (Weather5K)MiddleMiddleMiddle
C3S/CDS (M²Weather)HighHighHigh

Furthermore, compared with existing benchmarks such as Weather5K, in addition to our systematic validation of the importance of joint station-variable modeling, our benchmark also provides two additional contributions and insights:

  1. We correct misleading comparisons in the literature. Specifically, channel interactions in multivariate time-series models are typically designed to model relationships among different variables. When adapting such models, existing benchmarks have treated different stations as channels and used the same mechanism to model spatial relationships among stations, thereby conflating the two types of interactions in their comparisons, such as Corrformer in weather5k. Cross-variable dependencies and cross-station spatial dependencies are distinct modeling targets. To address this issue, we use model wrappers to distinguish the two types of interactions and fairly compare multi-station single-variable (MSSV) and single-station multi-variable (SSMV) models under a unified forecasting task and evaluation protocol.
  2. We propose a multi-criteria quality control and station stratification strategy. Weather5K selects stations based on a threshold of more than 90% valid hourly records. We additionally examine observation coverage for each variable, consecutive gaps, quality control pass rates, and station metadata consistency. This allows us to identify stations with high overall coverage but prolonged gaps for individual variables, and to select stations separately for the global and regional benchmarks.
AuthorsOpenReview ↗

Respond to Q3: Do the adapter results support end-to-end joint modeling of station and variable interactions?

Thank you for the review. To demonstrate that our adapter can improve forecasts beyond post-processing, we selected four backbone models of different types on the France dataset and compared three settings: the original backbone, fitting the adapter separately after freezing the backbone, and adding the adapter from the start of training and optimizing it jointly with the backbone. For all four models, both settings with the adapter outperform the corresponding original backbone. This shows that the complementary station and variable information introduced by the adapter can improve predictions both by correcting the outputs of frozen models and through end-to-end training. These results provide further empirical evidence for our main conclusion that jointly using station and variable interactions improves station weather forecasting performance.

ModelOriginal+ AdapterJoint Training
DLinear0.4979 ± 0.00050.4605 ± 0.00000.4698 ± 0.0001
S²Transformer0.3784 ± 0.00030.3657 ± 0.00010.3466 ± 0.0001
TQNet0.3703 ± 0.00040.3483 ± 0.00020.3513 ± 0.0004
HiSTGNN0.3089 ± 0.00440.3018 ± 0.00410.3011 ± 0.0056
Reviewer_abQfOpenReview ↗

Response to Authors

Thank you for the additional baseline, ablations, and joint-training experiments. These strengthen the evidence and address several of my concerns. However, my main concern about novelty and new scientific insight remains, consistent with Reviewer DzTY’s observation. Joint cross-station and cross-variable modeling has already been studied, and further confirming its benefits offers limited novelty. While I appreciate the improved quality control and unified evaluation protocol, I am not convinced that the benchmark’s incremental contribution meets the bar for ICLR. I therefore maintain my original assessment.

Reviewer_aXyPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces M²Weather, a benchmark for multi-station and multi-variable weather forecasting. It provides three datasets covering France, Europe, and the globe. The authors establish a unified evaluation framework and compare 16 forecasting models across four modeling paradigms. They also introduce a lightweight adapter to investigate the benefits of modeling interactions among weather stations and variables without retraining the original models. The experiments suggest that jointly modeling spatial and variable relationships can improve forecasting accuracy.

Critical strengths

1.Clear motivation and a unified benchmark The paper addresses the lack of standardized evaluation for multi-station and multi-variable weather forecasting. The unified framework enables systematic comparisons across different forecasting models and modeling paradigms.

2.Careful dataset construction and quality control The authors consider observation completeness, temporal consistency, data quality, and station metadata when selecting weather stations. This systematic process improves the reliability of the datasets and provides a useful foundation for future research.

3.Comprehensive model comparisons across spatial scales The benchmark evaluates 16 representative models across three spatial scales, covering different forecasting architectures and modeling paradigms. The additional adapter experiments provide supporting evidence for the benefits of using both station and variable information.

Critical weaknesses

1.Limited evaluation metrics for meteorological forecasting The benchmark mainly relies on MSE and MAE, which are insufficient to fully assess practical weather forecasting performance. In operational meteorology, forecast skill, such as ACC and CRPS, is an important evaluation criterion. Lower MSE or MAE alone does not necessarily indicate better forecasting skill. I suggest including forecast skill scores and evaluating performance across different forecast lead times to provide a more meaningful assessment.

2.Insufficient evaluation of the impact of missing-data imputation Although the overall imputation rates are relatively low, some variables have substantial missing values, such as station pressure in Europe (11.242%). Since the benchmark aims to compare forecasting models, it would be useful to verify whether model rankings remain consistent when evaluation is restricted to actual observations. The provided value-source masks should make this analysis straightforward.

3.Insufficient justification of the benchmark's advantages over existing benchmarks.

Although the paper introduces more detailed station quality control and multi-scale evaluation, it does not fully demonstrate how these designs improve upo1n existing weather forecasting benchmarks. For example, the authors could investigate whether their station selection strategy leads to more reliable or stable model rankings. In addition, the related work mainly focuses on station-based forecasting datasets and models, with limited discussion of broader meteorological benchmarks such as WeatherBench, RainBench, and WxC-Bench. While these benchmarks address different forecasting tasks, discussing their differences would help clarify the unique contributions and positioning of M²Weather.

4.Limited insights into future model development and forecasting challenges.

While M²Weather provides a unified comparison of different forecasting paradigms, its main finding that joint station-variable modeling improves performance is largely consistent with existing meteorological knowledge. The benchmark offers limited insights into when and why different interaction mechanisms are effective. Moreover, the forecasting task is restricted to fixed-station temporal prediction, without evaluating important challenges such as unseen-station generalization, cross-region transfer, or extreme-weather forecasting. Including these settings would make the benchmark more informative for future model development.

Minor comments: There are citation errors on pages 5 and 14.

Questions for authors

Please see weaknesses

6 comments
AuthorsOpenReview ↗

Respond to W1: How do models compare on forecast skill metrics such as ACC and CRPS?

Thank you for the suggestion. We agree with the reviewer that forecast skill is an important dimension of weather forecasting performance. We therefore report both the anomaly correlation coefficient (ACC) and the continuous ranked probability score (CRPS) at 48- and 72-hour lead times for representative models from the four modeling paradigms. Considering that in the case of deterministic prediction, CRPS is simply MAE, and to avoid retraining with an added variance head, we construct the predictive distribution using validation set residuals. HiSTGNN remains the leading model on the ACC and CRPS metrics, and after the missing interactions were completed, the results of every other model improved. This indicates that the benefits brought by jointly using station and variable information also extend to more forecast skill metrics. In the revised manuscript, we will report the ACC and CRPS results of more baselines and update the corresponding discussion.

Model ( lead times of 48 h)Original(ACC)↑+ Adapter(ACC)↑Original(CRPS)+ Adapter(CRPS)
SSSV / DLinear0.2560 ± 0.00070.2632 ± 0.00000.3701 ± 0.00030.3542 ± 0.0000
MSSV / S²Transformer0.3363 ± 0.00020.3473 ± 0.00070.3093 ± 0.00010.3035 ± 0.0000
SSMV / TQNet0.4117 ± 0.00020.4274 ± 0.00020.3043 ± 0.00010.2969 ± 0.0001
MSMV / HiSTGNN0.4575 ± 0.0030—0.2765 ± 0.0022—
Model ( lead times of 72h)Original(ACC)↑+ Adapter(ACC)↑Original(CRPS)+ Adapter(CRPS)
SSSV / DLinear0.3079 ± 0.00100.3114 ± 0.00000.3478 ± 0.00020.3432 ± 0.0000
MSSV / S²Transformer0.3582 ± 0.00050.3681 ± 0.00040.2760 ± 0.00010.2619 ± 0.0001
SSMV / TQNet0.4612 ± 0.00030.4759 ± 0.00030.2816 ± 0.00020.2758 ± 0.0002
MSMV / HiSTGNN0.4989 ± 0.0021—0.2540 ± 0.0009—
AuthorsOpenReview ↗

Respond to W2: Do model rankings remain consistent when evaluated only on observed values?

Thank you for the suggestion. To examine whether model rankings remain consistent on observed values, we selected one model from each of the four modeling paradigms and conducted an additional evaluation on the Europe dataset. We report MSE averaged across the four target variables. We excluded target values filled by interpolation or ERA5 from the error calculation and retained only observed values. The model that jointly models station and variable interactions still leads, and the relative ranking of the four models is consistent with the results in the main text that include imputed values. In the revised manuscript, we will report the results for all baselines evaluated only on observed values and update the corresponding discussion.

ModelMSE including imputed values (rank)MSE on observed values only (rank)
SSSV / DLinear0.4481 ± 0.0010 (4)0.4790 ± 0.0002 (4)
MSSV / S²Transformer0.3332 ± 0.0006 (2)0.3695 ± 0.0009 (2)
SSMV / TQNet0.3472 ± 0.0008 (3)0.3809 ± 0.0005 (3)
MSMV / HiSTGNN0.2908 ± 0.0031 (1)0.3192 ± 0.0037 (1)
AuthorsOpenReview ↗

Respond to W3: How does station screening affect rankings, and how does M²Weather compare with existing benchmarks?

We agree with the reviewer that it is necessary to examine how dataset construction strategies affect model evaluation. To test the effectiveness of multi-criteria quality control, we selected 100 stations from the same candidate station pool by random sampling and by multi-criteria quality control, respectively, and imputed missing values according to the method described in the paper. Subsequently, we compared the rankings of four types of models under two evaluation scopes: using only true observations, and including imputed values. The results show that the rankings of the four models in the multi-indicator screening group remained consistent, whereas the random sampling group exhibited rank changes. This demonstrates the effectiveness of our indicator quality control strategy.

ModelMulti-criteria quality control sampling / MSE including imputed valuesMulti-criteria quality control sampling / MSE on observed values onlyRandom sampling / MSE including imputed valuesRandom sampling / MSE on observed values only
SSSV / DLinear0.3485 ± 0.0001 (4)0.3559 ± 0.0000 (4)0.3031 ± 0.0000 (4)0.3577 ± 0.0000 (3)
MSSV / S²Transformer0.2882 ± 0.0005 (2)0.2940 ± 0.0006 (2)0.2538 ± 0.0006 (2)0.3380 ± 0.0004 (2)
SSMV / TQNet0.3247 ± 0.0001 (3)0.3315 ± 0.0001 (3)0.2799 ± 0.0001 (3)0.3708 ± 0.0002 (4)
MSMV / HiSTGNN0.2514 ± 0.0004 (1)0.2671 ± 0.0005 (1)0.2341 ± 0.0000 (1)0.3015 ± 0.0004 (1)

We have also added comparisons with other meteorological benchmarks to the related work section and below. WeatherBench primarily addresses global gridded weather forecasting, RainBench focuses on global precipitation forecasting, and WxC-Bench covers multimodal, multitask weather and climate applications. M²Weather focuses on multivariate forecasting over irregular station networks and systematically compares different paradigms for modeling station and variable interactions using station quality stratification, three spatial scales, and a unified evaluation protocol.

BenchmarkData typeMain taskCoverageEvaluation focus
WeatherBenchRegular gridMedium-range forecastingGlobalGridded forecast skill
RainBenchRegular gridPrecipitation forecastingGlobalPrecipitation forecast skill
WxC-BenchGrids / tracksWeather forecastingRegional / globalDownstream task performance
M²WeatherIrregular surface stationsMulti-station, multivariate forecastingRegional / globalSeparate and joint gains from station and variable interactions
Reviewer_DzTYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces M²Weather, a benchmark for multi-station and multi-variable station weather forecasting. It collects high-quality stations at three spatial scales through multi-criteria quality control and station stratification. It also sorts existing methods into four paradigms by whether they model station interactions and variable interactions, and compares them all under one protocol. On top of the benchmark, it proposes a lightweight adapter that adds the station or variable relationships a trained model misses without retraining the model. Across 16 models, joint modeling gives the lowest error, and the adapter lowers error for every model it is applied to.

Critical strengths
  • The description of the dataset construction is concrete. Station assessment combines long-term coverage, monthly and seasonal stability, consecutive gap lengths, quality control pass rates, hourly-alignment rates and identity conflict. Pre-imputation coverage and the proportion of stations with verified identity and current GBON affiliation are reported for each tier, and the percentage of values completed by station interpolation or ERA5 is reported for each dataset.
  • The paper designs lightweight, backpropagation-free post-processing adapters that complement the station and variable interactions missing from existing backbones. They can be added to a model that is already trained, so they may be useful in operational systems where rebuilding a model is costly.
Critical weaknesses
  • That multi-station and multi-variable modeling improves weather forecasting is common knowledge, so experiments that only confirm this direction are not very informative. If the aim is to systematically assess the individual and joint roles of station and variable interactions, the paper does not deliver this, since it has no ablation in which the same architecture is retrained without station interactions or without variable interactions. The adapter is only a linear mapping on standardized predictions, so it cannot capture non-linear station and variable interactions, and it is not a substitute for that ablation.
  • The comparison across paradigms is not a fair test. The model wrappers fold variables or stations into the batch dimension and share parameters, and the paper does not test whether this performs worse than training a separate MSSV model for each variable or a separate SSMV model for each station. MSMV has a single model, HiSTGNN, so its lead may reflect the model itself rather than joint modeling.
Questions for authors
  • Does the shared-parameter design lose accuracy compared with training one model per station or per variable?
  • Can you ablate HiSTGNN by removing station interactions and separately variable interactions, and report additional MSMV models?
  • Are the reported errors computed on observed values only, or do they also include values completed by interpolation and ERA5?
4 comments
AuthorsOpenReview ↗

Respond to Q1/W2: Does the shared-parameter design lose accuracy compared with training one model per station or per variable?

Thank you for the review. To examine how parameter sharing affects forecasting accuracy, we selected two MSSV models (STELLA and S²Transformer) and two SSMV models (TQNet and iTransformer). We randomly selected five stations and three variables (air temperature T, relative humidity RH, and station pressure P) from the France dataset. We compared MSE under shared-parameter training and separate training, with MSSV models trained separately for each variable and SSMV models trained separately for each station. We also added the adapter under both training settings to assess the gains from completing the missing interactions. The results show that training MSSV models separately for each variable yields lower MSE than shared-parameter training, while sharing parameters across stations yields lower MSE than training separate SSMV models for each station. Adding the adapter further reduces MSE under both training settings, providing further empirical evidence for the value of jointly using station and variable interactions. In practice, the choice of parameter sharing strategy can depend on the modeling paradigm and the trade-off between training cost and forecasting accuracy.

ModelTraining ParadigmOriginal(MSE)+ Adapter(MSE)
MSSVSTELLAShared parameter0.2627 ± 0.00030.2560 ± 0.0000
MSSVSTELLADedicated parameter0.2419 ± 0.00050.2394 ± 0.0002
MSSVS²TransformerShared parameter0.2759 ± 0.00050.2629 ± 0.0002
MSSVS²TransformerDedicated parameter0.2482 ± 0.00030.2330 ± 0.0004
SSMVTQNetShared parameter0.6519 ± 0.00130.6159 ± 0.0007
SSMVTQNetDedicated parameter0.6742 ± 0.00230.6179 ± 0.0016
SSMViTransformerShared parameter0.6506 ± 0.00080.5991 ± 0.0002
SSMViTransformerDedicated parameter0.6960 ± 0.00100.6257 ± 0.0056
AuthorsOpenReview ↗

Respond to Q2/W1: Can you ablate HiSTGNN by removing station interactions and separately variable interactions?

Thank you for the review. We conducted ablation experiments on HiSTGNN using the France dataset, removing variable interactions and station interactions separately. As shown in the table below, forecasting performance declined when either type of interaction was removed, indicating that both contribute to the forecasting performance of HiSTGNN.

HiSTGNN settingMSEMAE
Without variable interactions0.3098 ± 0.00480.3841 ± 0.0011
Without station interactions0.3150 ± 0.00010.3887± 0.0004
Full model0.3089 ± 0.00440.3736 ± 0.0014

We also report the result of the most recent MSMV method, StationPDE [1] (September, 2026) on the France dataset below. As we can see, StationPDE outperforms all SSSV, MSSV, and SSMV methods evaluated in the paper, providing further empirical evidence for the value of jointly modeling station and variable interactions. We will subsequently report StationPDE results on the Europe and Global datasets in the revised paper.

ModelMSEMAE
MSMV / StationPDE0.3062 ± 0.00140.3796 ± 0.0018

Furthermore, in addition to our systematic validation of the importance of joint station-variable modeling, our benchmark also provides two additional contributions and insights:

  1. We correct misleading comparisons in the literature. Specifically, channel interactions in multivariate time-series models are typically designed to model relationships among different variables. When adapting such models, existing benchmarks have treated different stations as channels and used the same mechanism to model spatial relationships among stations, thereby conflating the two types of interactions in their comparisons, such as Corrformer in weather5k. Cross-variable dependencies and cross-station spatial dependencies are distinct modeling targets. To address this issue, we use model wrappers to distinguish the two types of interactions and fairly compare multi-station single-variable (MSSV) and single-station multi-variable (SSMV) models under a unified forecasting task and evaluation protocol.
  2. We propose a multi-criteria quality control and station stratification strategy. Weather5K selects stations based on a threshold of more than 90% valid hourly records. We additionally examine observation coverage for each variable, consecutive gaps, quality control pass rates, and station metadata consistency. This allows us to identify stations with high overall coverage but prolonged gaps for individual variables, and to select stations separately for the global and regional benchmarks.

[1] Wang X, Chen C, Li R, et al. StationPDE: Station-Oriented Surface PDE Learning for Multi-Station Multivariate Weather Forecasting[J]. arXiv preprint arXiv:2609.22123, 2026.

AuthorsOpenReview ↗

Respond to Q3: Are the reported errors computed on observed values only, or do they also include values completed by interpolation and ERA5?

Thank you for the review. The original evaluation in the paper included target values filled using interpolation and ERA5. To examine whether our main conclusions still hold on observed values, we selected one model from each of the four modeling paradigms (DLinear, S²Transformer, TQNet, and HiSTGNN) and conducted an additional evaluation on the France dataset. We excluded all imputed target values from the error calculation and retained only observed values. HiSTGNN remained the leading model, and all three other models improved after the missing interactions were completed. This provides further evidence from observed values for the value of jointly modeling station and variable interactions.

ModelOriginal(MSE)+ Adapter(MSE)Original(MAE)+ Adapter(MAE)
SSSV / DLinear0.4990 ± 0.00050.4615 ± 0.00000.5161 ± 0.00050.4976 ± 0.0000
MSSV / S²Transformer0.3795 ± 0.00040.3666 ± 0.00010.4183 ± 0.00010.4129 ± 0.0001
SSMV / TQNet0.3709 ± 0.00040.3489 ± 0.00020.4146 ± 0.00020.4104 ± 0.0002
MSMV / HiSTGNN0.3092 ± 0.0044—0.3753 ± 0.0034—

Thank you for your response. The added StationPDE comparison and the evaluation on observed values address some of my concerns. However, I still worry that the training and evaluation protocol does not ensure a fair comparison across models, which is central to a benchmark paper. As the added experiments show, MSSV models benefit from dedicated parameters, which is not considered in the paper. Moreover, I share Reviewer abQf's concern about novelty. Therefore, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.