- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AirQualityBench introduces a global air-quality forecasting dataset assembled from OpenAQ, containing hourly observations from 3,720 stations over 2021–2025 for six pollutants. Its intended contributions are preserving observation missingness, evaluating forecasts in physical concentration scales, and comparing predictive accuracy with computational cost.
The benchmark uses 2021–2023 for training, 2024 for validation, and 2025 for testing. It supplies a geographic nearest-neighbor graph and evaluates twelve spatio-temporal models at forecasting horizons of 6, 12, 18, and 24 hours. Supplementary analyses include pollutant-specific results, graph-connectivity and mask ablations, station-coverage statistics, and seed robustness for D2STGNN.
Critical strengths
A useful dataset scope and forecasting setting. Combining five years of hourly observations from 3,720 stations across six pollutants offers a potentially valuable resource for studying forecasting under geographically uneven monitoring coverage. A full-year chronological test period is preferable to randomly mixing observations across time when assessing future forecasting performance.
Explicit attention to missing observations and computational cost. Preserving observation masks avoids treating synthetically completed future values as ground truth. The benchmark also recognizes that accuracy, memory, and latency matter jointly at this scale. The reported mask ablation provides preliminary evidence that observation-status information can be useful, although broader validation is needed.
Critical weaknesses
The numerical results contain contradictions that undermine confidence in the benchmark rankings. Several discrepancies cannot be reconciled with the stated metric definitions without additional explanation:
At the 6-hour horizon, Table 6 reports higher MAE for DCRNN than GWN for every pollutant: 3.14 versus 2.89 for PM2.5, 6.82 versus 6.14 for PM10, 4.12 versus 3.85 for NO₂, 4.56 versus 3.93 for O₃, 0.98 versus 0.86 for SO₂, and 51.23 versus 45.21 for CO. Nevertheless, Table 2 reports lower global MAE for DCRNN: 6.04 versus 6.09. Under the stated common evaluation mask and pooled MAE definition, the global value is a common non-negative weighted average of pollutant MAEs, so this ordering reversal should not occur.
Table 7 reports 12-hour CO MAE of 45.54 for STTN and 48.48 for D2STGNN, but Table 12 ranks D2STGNN first and STTN second for CO.
Equation 8 implies that, for a fixed pollutant and common evaluation set, zMAE equals MAE divided by that pollutant’s standard deviation. It must therefore preserve the within-pollutant ordering. However, Table 7 gives NO₂ MAE of 4.22 for STTN and 4.30 for GWN, while Table 11 reverses their ordering with zMAE of 0.63 and 0.62.
These discrepancies affect multiple tables and evaluation summaries. Because reliable measurement is the primary contribution of a benchmark paper, an audited reconciliation is necessary before its conclusions can be accepted.
Unharmonised units undermine the claimed physical interpretability and make aggregate rankings dependent on reporting conventions. Appendix F explicitly acknowledges heterogeneous provider units, particularly for CO, and leaves station-level unit auditing and harmonization to future work. Consequently, even a pollutant-specific MAE may combine errors expressed in different units. Separating pollutants does not by itself solve within-pollutant unit heterogeneity.
Calling the cross-pollutant aggregate a ranking statistic does not eliminate this issue: changing a provider’s unit convention rescales its contribution and can change model rankings without changing physical forecasting quality. Likewise, the MAPE cutoff of 0.1 in “original units” selects different physical thresholds under different unit conventions. Per-pollutant normalisation does not resolve heterogeneous units already mixed within that pollutant. Retaining original metadata is valuable, but a canonical-unit evaluation or explicitly unit-stratified protocol is needed to support the paper’s central claims.
Questions for authors
Numerical reconciliation: Can you regenerate Tables 2, 6, 7, 11, and 12 from the same saved predictions and masks, reporting the observation counts and exact aggregation weights? Please explain the DCRNN–GWN global-MAE reversal, the CO ranking discrepancy, and the NO₂ zMAE reversal identified above.
Unit handling: Which units occur for each pollutant, and how many stations and observations use each unit? Are different units pooled within the reported pollutant MAEs? Can you provide harmonized or unit-stratified results and assess whether the rankings change?
Reproducible protocol: What are the input-window length, forecast aggregation convention, normalisation fitting scope, quality-control thresholds, and model-specific mask adaptations? For computational profiling, please specify hardware, batch size, forecast horizon, numerical precision, timing procedure, and whether memory refers to training or inference.
Evidence for the central benchmark claim: Can you provide a controlled comparison of authentic and completed input histories while holding stations, forecast targets, and evaluation observations fixed? Also, how do persistence, seasonal persistence, and a simple non-graph forecasting model perform? These comparisons would help establish the benchmark’s difficulty and the specific contribution of its missingness protocol.
Selection and coverage: How do results change if station eligibility is determined using the training period only, rather than observation availability across 2021–2025? Can you report performance by region and missingness level? The current protocol describes forecasting at retrospectively selected resident stations, rather than establishing generalization to new or sparsely monitored regions.
Since the author has not yet replied I will maintain the same score and judgement.