acceptodds
Under review as a conference paper at ICLR 2027

AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting

Abstract

Air-quality forecasting models are commonly evaluated on regional, preprocessed, and normalized datasets, where missing observations are removed or artificially completed. Such protocols simplify comparison but hide the conditions that dominate real monitoring networks: uneven global coverage, structured missingness, heterogeneous pollutant scales, and deployment cost. We introduce AirQualityBench, a global multi-pollutant benchmark designed to evaluate forecasting models under these realistic conditions. The benchmark contains hourly observations from 3,720 monitoring stations over 2021–2025, covers six major pollutants, and preserves provider-native observation masks. Rather than imputing a dense data tensor, AirQualityBench exposes missingness as part of the forecasting problem and reports errors on valid future observations after inverse transformation to physical concentration scales. Evaluating representative spatio-temporal models under this unified protocol shows that strong performance on sanitized datasets does not reliably transfer to global, fragmented monitoring streams. AirQualityBench therefore serves as a realistic testbed for scalable, mask-aware, and physically interpretable air-quality forecasting. All benchmark data, code, evaluation scripts, and baseline implementations are available at https://anonymous.4open.science/r/AirQualityBench-85B1/.

Primary area:
datasets and benchmarks
Keywords:
Global Air Quality Forecasting, Mask-aware Spatio-temporal Benchmark, Physical-scale Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rc3zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AirQualityBench introduces a global air-quality forecasting dataset assembled from OpenAQ, containing hourly observations from 3,720 stations over 2021–2025 for six pollutants. Its intended contributions are preserving observation missingness, evaluating forecasts in physical concentration scales, and comparing predictive accuracy with computational cost.

The benchmark uses 2021–2023 for training, 2024 for validation, and 2025 for testing. It supplies a geographic nearest-neighbor graph and evaluates twelve spatio-temporal models at forecasting horizons of 6, 12, 18, and 24 hours. Supplementary analyses include pollutant-specific results, graph-connectivity and mask ablations, station-coverage statistics, and seed robustness for D2STGNN.

Critical strengths

A useful dataset scope and forecasting setting. Combining five years of hourly observations from 3,720 stations across six pollutants offers a potentially valuable resource for studying forecasting under geographically uneven monitoring coverage. A full-year chronological test period is preferable to randomly mixing observations across time when assessing future forecasting performance.

Explicit attention to missing observations and computational cost. Preserving observation masks avoids treating synthetically completed future values as ground truth. The benchmark also recognizes that accuracy, memory, and latency matter jointly at this scale. The reported mask ablation provides preliminary evidence that observation-status information can be useful, although broader validation is needed.

Critical weaknesses

The numerical results contain contradictions that undermine confidence in the benchmark rankings. Several discrepancies cannot be reconciled with the stated metric definitions without additional explanation:

At the 6-hour horizon, Table 6 reports higher MAE for DCRNN than GWN for every pollutant: 3.14 versus 2.89 for PM2.5, 6.82 versus 6.14 for PM10, 4.12 versus 3.85 for NO₂, 4.56 versus 3.93 for O₃, 0.98 versus 0.86 for SO₂, and 51.23 versus 45.21 for CO. Nevertheless, Table 2 reports lower global MAE for DCRNN: 6.04 versus 6.09. Under the stated common evaluation mask and pooled MAE definition, the global value is a common non-negative weighted average of pollutant MAEs, so this ordering reversal should not occur.

Table 7 reports 12-hour CO MAE of 45.54 for STTN and 48.48 for D2STGNN, but Table 12 ranks D2STGNN first and STTN second for CO.

Equation 8 implies that, for a fixed pollutant and common evaluation set, zMAE equals MAE divided by that pollutant’s standard deviation. It must therefore preserve the within-pollutant ordering. However, Table 7 gives NO₂ MAE of 4.22 for STTN and 4.30 for GWN, while Table 11 reverses their ordering with zMAE of 0.63 and 0.62.

These discrepancies affect multiple tables and evaluation summaries. Because reliable measurement is the primary contribution of a benchmark paper, an audited reconciliation is necessary before its conclusions can be accepted.

Unharmonised units undermine the claimed physical interpretability and make aggregate rankings dependent on reporting conventions. Appendix F explicitly acknowledges heterogeneous provider units, particularly for CO, and leaves station-level unit auditing and harmonization to future work. Consequently, even a pollutant-specific MAE may combine errors expressed in different units. Separating pollutants does not by itself solve within-pollutant unit heterogeneity.

Calling the cross-pollutant aggregate a ranking statistic does not eliminate this issue: changing a provider’s unit convention rescales its contribution and can change model rankings without changing physical forecasting quality. Likewise, the MAPE cutoff of 0.1 in “original units” selects different physical thresholds under different unit conventions. Per-pollutant normalisation does not resolve heterogeneous units already mixed within that pollutant. Retaining original metadata is valuable, but a canonical-unit evaluation or explicitly unit-stratified protocol is needed to support the paper’s central claims.

Questions for authors

Numerical reconciliation: Can you regenerate Tables 2, 6, 7, 11, and 12 from the same saved predictions and masks, reporting the observation counts and exact aggregation weights? Please explain the DCRNN–GWN global-MAE reversal, the CO ranking discrepancy, and the NO₂ zMAE reversal identified above.

Unit handling: Which units occur for each pollutant, and how many stations and observations use each unit? Are different units pooled within the reported pollutant MAEs? Can you provide harmonized or unit-stratified results and assess whether the rankings change?

Reproducible protocol: What are the input-window length, forecast aggregation convention, normalisation fitting scope, quality-control thresholds, and model-specific mask adaptations? For computational profiling, please specify hardware, batch size, forecast horizon, numerical precision, timing procedure, and whether memory refers to training or inference.

Evidence for the central benchmark claim: Can you provide a controlled comparison of authentic and completed input histories while holding stations, forecast targets, and evaluation observations fixed? Also, how do persistence, seasonal persistence, and a simple non-graph forecasting model perform? These comparisons would help establish the benchmark’s difficulty and the specific contribution of its missingness protocol.

Selection and coverage: How do results change if station eligibility is determined using the training period only, rather than observation availability across 2021–2025? Can you report performance by region and missingness level? The current protocol describes forecasting at retrospectively selected resident stations, rather than establishing generalization to new or sparsely monitored regions.

1 comment

Since the author has not yet replied I will maintain the same score and judgement.

Reviewer_WmaWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • AirQualityBench evaluates forecasting from incomplete air-quality monitoring streams.

    • The dataset contains hourly observations from 3,720 stations over 2021–2025 for six pollutants. It preserves observation masks and evaluates predictions only where valid future measurements are available. The chronological split uses 2021–2023 for training, 2024 for validation, and 2025 for testing.
    • The task predicts future concentrations at the existing station locations from historical observations and masks, with a geographic graph as the spatial prior. It does not evaluate a spatially complete concentration field or prediction at arbitrary unmonitored locations.
  • The contribution is a dataset and evaluation protocol rather than a new forecasting model.

    • The paper compares twelve spatio-temporal models across four horizons and reports pollutant-wise errors, computational costs, and supplementary graph and masking ablations.
    • Its central claim is that a globally distributed dataset with authentic missingness provides a more deployment-relevant assessment than completed regional datasets.
Critical strengths
  • The benchmark addresses an important limitation of completed observation datasets.

    • Preserving the original observation masks avoids treating interpolated target values as actual measurements. This is a useful starting point for comparing forecasting methods under incomplete sensing.
    • The five-year, multi-pollutant collection and full-year test period provide a potentially useful resource beyond a single regional dataset. Reusing OpenAQ is not itself a limitation if the resulting curation and protocol enable reliable, reproducible evaluation.
  • The paper considers more than aggregate predictive accuracy.

    • Pollutant-wise results, inference cost, graph connectivity, and seed robustness are useful components of the evaluation. The masking ablation also reports a deterioration in D2STGNN's 12-hour MAE from 6.62 to 7.45, although what this ablation removes needs clarification (Weakness 2).
    • The authors explicitly acknowledge regional imbalance and provider-unit heterogeneity. This transparency is helpful, although the latter issue needs to be resolved before the benchmark can support physically interpretable comparisons.
Critical weaknesses
  1. The reported scores and rankings are not sufficiently reliable.

    • Appendix F acknowledges that provider-native units have not been fully harmonized. Calling the global aggregate a ranking statistic does not resolve this issue. Changing a unit convention changes the relative weight of a pollutant or provider and can change model rankings without changing the underlying predictions. Even pollutant-wise errors remain difficult to interpret when units differ within the same pollutant.
    • There are direct inconsistencies between the tables. At 12 hours, Table 7 reports lower CO MAE for STTN than D2STGNN, 45.54 versus 48.48, but Table 12 ranks D2STGNN first. Table 12 also assigns rank 7 to both PDFormer and DSTAGNN in every column. In addition, all six pollutant MAEs of AGCRN increase from 18 to 24 hours (Tables 8 and 9), yet its global MAE in Table 2 falls from 12.24 to 7.59. These need to be corrected from the underlying predictions.
    • The zMAE results also conflict with the stated definition. With a shared pollutant-specific standard deviation and the same evaluation mask, zMAE equals MAE divided by that standard deviation and cannot reverse the within-pollutant ordering. However, STTN has lower NO₂ MAE than GWN, 4.22 versus 4.30, but higher zMAE, 0.63 versus 0.62. If different aggregation or normalization procedures were used, they must be specified. Otherwise, the scores need to be regenerated.
  2. The released implementation raises questions about the masking protocol and run settings.

    • In the released code, the observation mask does not appear to be passed to the models as an input. Missing inputs are set to zero after normalization, which corresponds to filling them with the normalization mean. The paper states that masks are provided as additional input indicators when applicable. If the released code reflects the reported runs, the baselines are effectively mean-imputed models. The authors should clarify which models receive the input mask and what the masking ablation removes.
    • The released code averages metrics over test batches, and the runner scripts use batch sizes from 2 to 64. Most runs use a sliding-window step equal to the horizon, but AGCRN, GWN, and STGCN at 24 hours and DCRNN at 6 hours use a step of 1. It is unclear whether the reported results used these settings. If they did, the models were trained and evaluated on different window sets, and this needs explanation. Some of these runs coincide with the irregular entries above, such as AGCRN at 24 hours.
    • The MAPE scaling and graph weighting in the code also differ from Equations 5 and 7. Please confirm which versions produced the reported numbers.
  3. The sparse and uneven station coverage is not matched by sufficient spatial or deployment-oriented validation.

    • Although 3,720 stations is a substantial monitoring network, its distribution is highly uneven for a global benchmark. Approximately 84.2% of stations are in Asia, Europe, and North America, while Africa has 187 and Oceania has 90. The total count also overstates the coverage available for individual pollutants, since CO has only 1,456 active stations.
    • The temporal split evaluates future observations at the existing station network. It does not establish performance at unseen stations, unmonitored locations, or times without target observations. Station-based forecasting is a legitimate task, and I do not require a gridded simulator. However, the current evidence supports a narrower claim than broadly reliable forecasting across sparsely monitored regions.
    • The benchmark needs performance breakdowns by region, input-gap duration, pollution severity, season, and local station density, with sample counts and appropriate uncertainty estimates. Nearest-neighbor distances and graph-edge lengths would also help explain what the geographic graph represents in dense and sparse regions.
  4. The benchmark's additional practical value is not established by the current baseline comparison.

    • The twelve baselines are predominantly traffic-oriented spatio-temporal models. The evaluation lacks persistence and seasonal-naïve references, as well as representative air-quality-specific and time-series foundation models. The paper discusses AirFormer and PCDCNet but does not include them in the reported comparison.
    • A comparison with Aurora's air-pollution model or CAMS would help contextualize practical performance. These should be separate external-information references rather than nominally matched observation-only baselines. Aurora requires CAMS atmospheric fields and additional variables, so the comparison must disclose the information difference and align forecast issue times, pollutants, units, and station locations. Neither system should be treated as ground truth.
    • A focused comparison between mask-aware and input-imputed approaches, with the same genuinely observed test targets, would help demonstrate what the proposed protocol reveals that a conventional evaluation misses. This comparison is especially important given the question raised in Weakness 2.
  5. The related-work coverage could be improved.

    • The positioning should account for recent multi-region forecasting benchmarks such as Air Quality Arena (Bharadwaj et al., 2026) and distinguish the proposed observation-only setting from atmospheric foundation models and operational forecasting systems. I view the literature omissions as secondary to the evaluation and validation concerns above.
Questions for authors
  • Metric definitions and result consistency

    • Can the authors provide an audited unit mapping, explicit input and target mask definitions, and the exact aggregation procedure used for every reported metric?
    • Which models receive the observation mask as input, and what does the masking ablation remove? Were the batch sizes and window steps in the released runner scripts used for the reported results?
    • Please regenerate Tables 2 and 6 to 12 from a consistent set of predictions and explain the CO ranking discrepancy, the NO₂ MAE and zMAE reversal, and the AGCRN 24-hour entry. Do the principal rankings and conclusions remain unchanged?
  • Spatial coverage and deployment scope

    • What evidence shows that the aggregate results remain informative in sparsely monitored regions? Please report performance by region and local station density, together with the distribution of geographic neighbor distances.
    • How does performance change with longer input gaps, high-pollution events, and seasonal conditions? If the claims extend to new locations, spatially held-out evaluation would be needed. Otherwise, please state explicitly that the benchmark targets future forecasting at the existing station network.
  • Baseline adequacy and incremental benchmark value

    • How do persistence, seasonal-naïve, a representative air-quality model, and a representative time-series foundation model perform under the same observation-only protocol?
    • Can Aurora or CAMS provide a separately labeled external-information comparison on compatible targets? What model differences or failure modes does authentic-mask evaluation reveal that are missed when only the historical inputs are imputed and the observed targets are held fixed?

  • The dataset direction is useful, but the current submission does not yet provide sufficiently reliable scores or enough evidence for its claimed deployment relevance. For a datasets-and-benchmarks contribution, these are central requirements.
  • The first priority is to resolve the unit, masking, aggregation, and table inconsistencies, and to clarify how the released code relates to the reported runs. The next priorities are stronger reference baselines and performance analyses under sparse, incomplete, and high-pollution conditions. I would reconsider the assessment if these establish a reliable benchmark with a clear contribution beyond consolidating existing observations.
Reviewer_2KL2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces AirQualityBench, a global multi‑pollutant benchmark for air‑quality forecasting built from OpenAQ hourly observations spanning 2021‑2025 across 3,720 monitoring stations. Unlike existing regional, imputed benchmarks, it preserves authentic structured missingness and mandates physical‑unit evaluation instead of normalized metrics. It defines a masked spatio‑temporal forecasting task, provides a standardized spherical‑k‑NN graph accounting for earth curvature, chronological train‑val‑test splits, and a suite of baseline implementations covering graph‑RNN, temporal‑conv, transformer‑style spatio‑temporal models. Extensive experiments report pollutant‑wise forecasting accuracy plus inference memory/latency, exposing accuracy‑efficiency trade‑offs under real‑world fragmented sensing conditions. Results demonstrate sanitized‑dataset performance does not reliably transfer to global realistic settings.

Critical strengths

This work provides a complete, reproducible benchmark package with thorough multi‑dimension evaluation. The release includes standardized graph construction, fixed chronological splits, open‑source baseline implementations, unified evaluation scripts. Beyond forecasting accuracy, it explicitly profiles computational cost and analyzes accuracy‑efficiency trade‑offs. It also conducts diagnostic analysis of spatial correlation decay and pollutant‑specific missingness, offering rich diagnostics rather than only leaderboard numbers.

Critical weaknesses

Atmospheric pollutant transport and dispersion are governed by meteorological dynamics (e.g., wind speed/direction, boundary layer height, temperature, precipitation, and relative humidity). Restricting the benchmark purely to autoregressive historical pollutant concentrations and a static spatial graph divorces the forecasting problem from fundamental physical drivers. This limits the benchmark's ability to evaluate physics-informed or meteorology-conditioned models that represent the state-of-the-art in environmental science.

Questions for authors

How can the proposed methods/benchmark be applied to real‑world scenarios in the absence of terrain, landform, meteorological and other auxiliary information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.