acceptodds
Withdrawn from ICLR 2027

AirportMIST: A Multimodal Benchmark for Airport Weather-State Forecasting with Rare Adverse Conditions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Airport operations depend on timely forecasts of operationally meaningful local states, including rare adverse conditions such as thunderstorms and degraded flight conditions, yet existing weather and aviation benchmarks primarily focus on continuous variables, gridded atmospheric fields, or individual hazards. We introduce AirportMIST, a multimodal benchmark for multi-airport, multi-horizon categorical airport weather-state forecasting. AirportMIST covers 37 major civil airports in China during JJA 2020 and JJA 2025 and integrates high-frequency AWOS observations, irregular METAR reports, regional WAFS forecast fields in GRIB format, and static airport metadata. It defines precipitation/thunderstorm and flight-condition forecasting at 6-, 12-, and 24-hour horizons, together with chronological splits, causality-preserving multimodal alignment, and rare-event evaluation. We further provide MISTNet as a reproducible reference model for heterogeneous local and regional inputs. Experiments with operational, statistical, and time-series baselines reveal that rare-event ranking, multiclass argmax detection, and operational forecast behavior can lead to different conclusions: several learned models retain nonzero AP while making no Thunder predictions under multiclass argmax, whereas risk-aware TAF forecasts operate at a substantially higher predicted-positive rate. These results motivate evaluating airport weather-state forecasting beyond aggregate classification performance. AirportMIST provides a standardized and reproducible foundation for research on multimodal airport weather-state forecasting and rare-event prediction. Our code is available at https://anonymous.4open.science/r/MIST-1FA1/.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, multimodal forecasting, airport weather, rare-event prediction, spatiotemporal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KiFzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines automated airport observations, aviation weather reports, regional numerical forecast fields, and airport metadata. It defines precipitation and thunderstorm-state and flight-condition classification at several forecast horizons, then evaluates a reference model alongside learned and operational forecast baselines using overall classification, rare-event ranking, and categorical decision measures.

Critical strengths

Separating rare-event ranking from decisions provides a useful experimental view. In the main results table, some learned models retain a Thunder ranking signal but never predict Thunder as their final class, while the operational forecast baseline predicts adverse events at a markedly different frequency. Reporting these behaviors together reveals differences that an aggregate classification measure alone would hide.

Critical weaknesses

In “Anchor Construction and Causal Alignment,” a GRIB record is eligible when its initialization time is no later than the forecast anchor; the adjoining data description calls this nominal-cycle causal selection. Initialization time need not coincide with the time at which the forecast file was actually published or received, so this rule alone does not establish that the selected field was available at the anchor. For TAF, by contrast, the paper explicitly uses the later of issue and archive-arrival times to establish availability. If a GRIB field became available after the anchor, results using that modality could incorporate information unavailable when the prediction was made. The present description does not rule this out, leaving the benchmark's forecast-time causality and the associated performance interpretation uncertain.

In “Input Adaptation for Learnable Baselines,” the neural time-series baselines reduce each GRIB channel around an airport to a spatial mean, and the appendix explicitly notes that the grid's internal spatial arrangement is not retained. MISTNet instead keeps spatial grid tokens and aggregates them conditionally. The ablations show that regional input and fusion within MISTNet matter, but they cannot distinguish whether the gap in the main comparison comes from the fusion design or from spatial information discarded before the baselines receive it. Claims about the reference model's advantage over other learned methods therefore need to be confined to the present input adaptations.

Questions for authors

Could the authors clarify how the actual publication or archive-arrival times of the GRIB fields relate to each forecast anchor, and verify whether the causal protocol and main results hold when records are selected by actual availability time?

Could the authors provide a learned baseline that retains the GRIB spatial arrangement under comparable input and training conditions, or clarify how the main model comparison should be interpreted given the current spatial-mean adaptation?

Reviewer_sV4JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a multimodal benchmark for airport weather-state forecasting with rare adverse conditions. Specifically, this benchmark consists of 37 major civil airports in China during JJA 2020 and JJA 2025. Besides, it integrates a lot of variables, like high-frequency AWOS observations, irregular METAR reports, and regional WAFS forecast fields and etc. In addition, the authors also propose a reference model, MISTNet, to examine the applicability of this benchmark.

Critical strengths
  1. This is a valuable dataset for forecasting airport weather conditions. It covers a large number of airports and includes a rich set of variables, making it suitable for forecasting in real-world operational settings. It provides important support for airport operations and broader societal decision-making.
Critical weaknesses
  1. The descriptions of the variables in the benchmark are still somewhat high-level and abstract. For example, general readers may be unfamiliar with AWOS and METAR, which appear in Figure 1. The data shapes, value ranges, and specific physical meanings are unclear. I suggest that the authors present this information in a more accessible way and provide concrete examples where possible.
  2. The description of MISTNet also lacks detail. The architecture diagram in Figure 2 is too high-level and abstract: it is unclear what the inputs and outputs of MISTNet are, what their shapes are, and how the network operates internally. Although Figure 2 contains a large amount of information, I find that much of it does little to clarify the method, while essential details are scarce. Specifying tensor shapes at appropriate points in the description of the network architecture would make it easier to understand.
  3. As shown in Table 11, the ablation study examines at least six components. I encourage the authors to streamline the presentation and highlight the key design choices, as the current presentation makes it difficult to identify the most important designs and contributions.
Questions for authors
  1. One of my key concerns is whether the dataset can be publicly released. A major contribution of this work may lie in the value of the dataset, which, if used effectively, could help improve real-world operations.
Reviewer_P4yQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AirportMIST, a multimodal benchmark for airport weather-state forecasting across 37 airports in China. It integrates AWOS observations, METAR reports, WAFS forecast fields, and airport metadata to support precipitation/thunderstorm and flight-condition forecasting at 6-, 12-, and 24-hour horizons. The authors also propose MISTNet as a reference model and evaluate it against operational, statistical, and time-series baselines, highlighting the differences between rare-event ranking and actual event detection.

Critical strengths

The benchmark addresses a practically relevant forecasting problem and integrates multiple complementary data sources with standardized labels, chronological splits, and rare-event evaluation.

The experimental analysis provides useful insights into class imbalance, temporal distribution shifts, and the differences between ranking performance and argmax-based event detection. The paper also includes cross-airport evaluation and implementation details.

Critical weaknesses

Novelty relative to existing benchmarks needs stronger justification. The paper discusses AeroWF and other related resources, but does not provide a sufficiently clear, systematic comparison of their coverage, modalities, forecasting tasks, and evaluation protocols. The incremental contribution of AirportMIST over existing airport-weather benchmarks therefore remains insufficiently established.

The evaluation scope is limited. The benchmark covers 37 Chinese airports and only two summer periods, with testing restricted to August 2025. This does not establish robust generalization across independent years or seasons. Moreover, the leave-airports-out evaluation focuses only on precipitation/thunderstorm forecasting and does not compare MISTNet against a broader set of strong baselines.

Baseline fairness and causal alignment require further clarification. The generic time-series baselines use an adapter that spatially averages GRIB fields and interpolates irregular METAR reports, potentially discarding useful spatial and temporal structure. Stronger multimodal baselines or direct comparisons with relevant airport-weather models would help establish the advantages of MISTNet. In addition, the GRIB alignment rule is based on initialization/reference time, but it is unclear whether actual product availability time is also considered when ensuring forecast-time causality.

Data availability remains insufficiently concrete. The processed benchmark is promised upon acceptance, subject to applicable licenses and access conditions. Since the dataset itself is a central contribution, the authors should clarify whether all essential data and supporting resources can be publicly accessed and reproduced.

Questions for authors

Could the authors provide a systematic comparison with AeroWF and other relevant airport-weather benchmarks, and clarify the specific advances offered by AirportMIST?

Could the authors justify the baseline input adaptations and include stronger multimodal comparisons? Can the temporal evaluation be extended to an independent year or season?

Does GRIB selection account for the actual product issue/availability time, rather than only initialization time? What is the concrete plan for public dataset access and reproducibility?

I am open to revising my rating upward or downward depending on the authors' rebuttal and how satisfactorily these concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.