- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work establishes a novel national benchmark for predicting initial attack (IA) failures from publicly available environmental and contextual data under a discovery-day information contract. The benchmark aligns 38,128 naturally ignited FPA-FOD forest fire incidents with FIRMS/VIIRS thermal anomaly detections, gridMET weather and fire-danger variables, LANDFIRE vegetation, fuel, and terrain layers, OpenStreetMap accessibility features, and WorldPop population data. Under a single evaluation protocol, it benchmarks 16 representative models in terms of tables, time, space, and the family of spacetime.
Critical strengths
-
It builds a novel national benchmark to predict IA failures based on publicly available environmental and contextual data under the discovery day information contract.
-
This work may be useful to the community due to the contract. The wildfire machine learning literature is filled with work that leaks results implicitly, but this paper clearly defines event units, label rules, time sequence division, prohibited feature lists and alignment rules between discovery days, and explains the reasons for each prohibition, establishing a referable standard for this direction.
-
The experiments are meaningful, with 16 baselines across four representation families and six classification metrics, including Brier and ECE.
-
The work releases preprocessing output, model-ready cache, configuration, checkpoints, and an end-to-end reproducible path has also been provided.
Critical weaknesses
-
The LANDFIRE year used for each fire year should be indicated. It should be ensured that the disturbed year is earlier than the discovery date; the same applies to the OSM snapshot. If leakage is confirmed, a new experiment is required.
-
The paper takes event-level triage as the application scenario, but the "utility" has not been compared with any real-world baseline: neither simple heuristic rules such as fire risk index thresholds were compared, nor physical spread models like FARSITE/FlamMap were compared, nor were decision analysis metrics provided.
-
The generalization ability is insufficient. Only a single test year (2020, an abnormal year for fire activity) was used, and no spatial separation experiments were set up.
Questions for authors
-
The LANDFIRE year used for each fire year should be indicated. It should be ensured that the disturbed year is earlier than the discovery date; the same applies to the OSM snapshot. If leakage is confirmed, a new experiment is required.
-
The paper takes event-level triage as the application scenario, but the "utility" has not been compared with any real-world baseline: neither simple heuristic rules such as fire risk index thresholds were compared, nor physical spread models like FARSITE/FlamMap were compared, nor were decision analysis metrics provided.
-
The generalization ability is insufficient. Only a single test year (2020, an abnormal year for fire activity) was used, and no spatial separation experiments were set up.
3 comments
Thanks for the response. The response addressed some concerns. However, the added generalization evaluation seems to be trained on a new training and validation dataset configuration, which did not use the same setup. It is better to test with the same model on different years. I will consider other reviewers' comments for further consideration. Thank you.
Response: Generalization across test years.
We thank the reviewer for the helpful clarification and suggestion to evaluate the same models across different test years. We have conducted additional experiments and updated the revised manuscript accordingly.
Our original benchmark uses 2016–2018 for training, 2019 for validation, and 2020 for testing. This chronological evaluation follows common practice in wildfire prediction studies, where the most recent available years are held out for testing [1, 2]. At the time of our original experiments, the FPA-FOD edition we used only covered events through 2020, which limited our evaluation to that test year.
Fortunately, an updated FPA-FOD edition was released in 2026, extending coverage through 2024. We have since processed the additional 2021–2024 events. The following table reports AUPRC / AUROC (%) as mean ± standard deviation across five seeds.
| Test year | Events | Positive rate | XGBoost | Transformer | Swin-UNet | SwinLSTM |
|---|---|---|---|---|---|---|
| 2020 | 6,168 | 8.3% | 53.3±0.3 / 87.1±0.1 | 50.6±0.3 / 84.0±0.4 | 51.1±1.2 / 85.3±0.6 | 51.7±1.2 / 85.7±0.9 |
| 2021 | 6,528 | 6.2% | 43.8±0.4 / 86.0±0.2 | 43.7±0.3 / 84.8±0.4 | 38.0±1.5 / 82.8±0.7 | 39.7±0.8 / 82.7±0.8 |
| 2022 | 6,515 | 4.5% | 39.4±0.4 / 85.8±0.1 | 37.1±0.3 / 82.3±0.7 | 31.9±1.6 / 81.9±0.7 | 38.9±1.0 / 86.2±0.9 |
| 2023 | 6,367 | 5.6% | 41.8±0.3 / 86.4±0.2 | 41.2±0.3 / 83.2±0.4 | 34.7±1.5 / 80.9±1.0 | 38.1±0.6 / 83.7±0.4 |
| 2024 | 7,236 | 7.4% | 44.5±0.4 / 86.7±0.3 | 43.5±0.4 / 85.2±0.3 | 39.0±0.5 / 81.9±1.1 | 42.2±1.1 / 83.6±0.6 |
The original 2020 test year achieves the highest AUPRC, with XGBoost reaching 53.3%. Although AUPRC is lower in 2021–2024, these years also have lower positive-class prevalence (4.5–7.4%, compared with 8.3% in 2020), which can affect AUPRC. Importantly, XGBoost maintains relatively stable AUROC (85.8–87.1%) across all five years, suggesting that its discrimination ability is not limited to the unusual 2020 fire season. We acknowledge that performance varies across years, particularly for the spatial models.
We have incorporated this additional temporal generalization experiment and its results into Appendix C, Table 12 of the revised manuscript.
References [1] Kondylatos et al. Mesogeos: A Multi-Purpose Dataset for Data-Driven Wildfire Modeling in the Mediterranean. NeurIPS, 2023. [2] De Rango et al. CalWildFire: A High-Resolution In Situ Dataset for Wildfire Analysis and Modeling in the Mediterranean Region. Scientific Data, 2026.
Respond to Reviewer RXBj
Thank you for the thoughtful review. We have updated the PDF and addressed your concerns as below:
We use LF2016 for fuel, vegetation, and disturbance features, and LF2020 only for topography, including elevation, slope, and aspect. LF2016 includes disturbances through 2016, while some fuel layers reflect modeled recovery toward 2019–2020 [1]. These layers may therefore contain information unavailable at the time of earlier fires. However, elevation, slope, and aspect are relatively stable terrain attributes that LANDFIRE rarely updates [2]. We therefore use LF2020 topography for wildfire events from 2016 to 2020.
For OpenStreetMap (OSM), we address historical alignment by specifying snapshots from January 1 of each fire year. These snapshot dates precede the corresponding fire discovery dates. We now clarify these source choices and their timing limitations in Section 2.2.
To assess whether our results depend on these sources despite the timing limitations, we conducted an additional ablation. We first excluded all LANDFIRE features and then also excluded OSM, keeping all other inputs and evaluation settings unchanged. Five-seed test AUPRC (%, mean±std):
All four models retain predictive signal after both exclusions. XGBoost achieves 48.6% AUPRC compared with 53.3% under full input. These results show that prediction does not depend entirely on LANDFIRE and OSM, although historical timing concerns remain for the full-source setting. Further details are provided in Appendix C, Table 17.
To compare our models with established fire-danger measures, we added Energy Release Component (ERC) and Burning Index (BI) baselines. We used the same 6,168 test fires, including 513 positive cases, and measured ranking and recall at fixed review budgets. Results (%):
At a 5% review budget, XGBoost retrieves 38.0% of positive cases, compared with 9.9% for ERC and 12.3% for BI. These results support better event-level ranking under our protocol. Actual dispatch benefits remain untested. FARSITE and FlamMap simulate fire growth [3,4]. A direct comparison would require simulation inputs, a prediction horizon, and a rule linking simulated fire behavior to our labels. Further details are in Appendix C, Table 13.
We agree that one test year cannot establish performance across seasons. To assess test-year sensitivity, we added a forward-time evaluation using 2016–2017 for training, 2018 for validation, and 2019 for testing. We compare it with the original 2020 test. Five-seed test AUPRC/AUROC (%, mean±std):
The additional evaluation shows consistent predictive performance across two test years. XGBoost achieves the highest AUPRC in both years, with similar AUROC (87.3% in 2019 and 87.1% in 2020). The lower 2019 AUPRC (45.9% versus 53.3%) also reflects differences in positive rates (6.1% versus 8.3%) and training periods. These results provide further evidence of the benchmark's predictive value across fire seasons.
Both evaluations use chronological splits across U.S. wildfire events. This design assesses performance across years, while a geographically disjoint split would further assess transfer to unseen regions. We report the full five-seed results in Appendix C, Table 12.
References [1] LANDFIRE. LF2016 Remap Project Closeout Report. LANDFIRE. [2] LANDFIRE. Topographic Products. LANDFIRE. [3] Finney. FARSITE: Fire Area Simulator–Model Development and Evaluation. USDA Forest Service Research Paper RMRS-RP-4, 1998. [4] USDA Forest Service. FlamMap. Fire, Fuel, and Smoke Science Program.