acceptodds
Under review as a conference paper at ICLR 2027

FireResBench: An Event-Centered Benchmark for Wildfire Lifecycle Understanding and Operational-Response Forecasting

Abstract

Wildfire emergency management requires responders to continuously assess evolving fire behavior, incident status, containment progress, and resource deployment using scattered and irregular operational records. Existing wildfire datasets have largely been designed for physical fire detection, burned-area mapping, or spread forecasting, leaving evolving incident observations disconnected from real-world response histories. To bridge this gap, we introduce **FireResBench**, an event-centered benchmark that connects wildfire progression with emergency-response records by organizing operational records into temporally ordered Fire-Day histories. FireResBench supports two complementary tasks: (1) wildfire lifecycle-state understanding and (2) daily personnel and cost forecasting. To provide fire lifecycle supervision, a two-stage labeling paradigm is adopted. In the first stage, carefully designed labeling rules guide an auto-labeling code pipeline that generates provisional labels. In the second stage, domain scholars verify and correct the labels to obtain the final annotations. The released dataset covers the conterminous United States from January 2017 to December 2020 and contains 33,303 Fire-Day instances from 5,728 incidents. It provides scholar-verified lifecycle labels, temporally valid daily personnel and cost targets, incident-disjoint splits, and carefully designed evaluation protocols. Experiments across representative machine learning, tabular, and deep learning paradigms reveal that accurate forecasting on lifecycle transitions and high-demand Fire-Days remain particularly challenging. By converting fragmented public records into auditable event sequences and reproducible learning tasks, FireResBench establishes a foundation for data-driven wildfire-response modeling. The benchmark and evaluation resources are available at: https://anonymous.4open.science/r/FireResBench-0A9D/.

Primary area:
datasets and benchmarks
Keywords:
Wildfire Emergency Response, Benchmark Datasets, Wildfire Lifecycle Understanding, Operational Response Forecasting, Machine Learning for Disaster Management

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qCryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FireResBench, an event-centered benchmark for wildfire lifecycle understanding and operational-response forecasting. The benchmark organizes 2017–2020 wildfire incident records from the CONUS into 33,303 Fire-Day instances covering 5,728 incidents. It defines two tasks: (1) predicting the current wildfire lifecycle state from five operational phases, with labels constructed through rule-based labeling followed by domain-scholar review; and (2) forecasting next-day personnel demand and daily cost from temporally valid incident histories. The paper evaluates a broad set of tree-based, tabular-attention, recurrent, and temporal forecasting models. Beyond aggregate metrics, the experiments analyze lifecycle boundaries, demand tails, incident-level response trajectories, feature contributions, historical-context length, temporal OOD generalization, and missingness robustness. The results show that FT-Transformer is generally the strongest baseline, while errors concentrate around intermediate lifecycle boundaries and high-demand operational regimes. Overall, the benchmark addresses a relatively underexplored operational aspect of wildfire modeling, but the empirical validation of label quality and the difficulty/validity of some forecasting targets could be strengthened.

Critical strengths
  1. The paper addresses a meaningful and relatively underexplored problem. FireResBench moves beyond conventional wildfire benchmarks focused on detection, spread, or burned area by explicitly modeling operational lifecycle states and response-resource demand. The event-centered Fire-Day representation and strict temporal evidence boundary are sensible design choices for this setting. 2. The evaluation is broader than a standard benchmark comparison. In addition to aggregate performance, the paper investigates lifecycle transitions, demand tails, incident-level trajectories, feature ablations, historical context, temporal OOD generalization, and missingness. These analyses provide useful evidence about where current models succeed and fail.
Critical weaknesses
  1. The validity of the Task A lifecycle labels is not sufficiently quantified. The paper describes a two-stage labeling procedure combining rule-based labeling and domain-scholar review, but it does not report key annotation-quality statistics, such as inter-annotator agreement, correction/defer rates, or the proportion of automatically assigned labels that were modified during expert review. Therefore, it remains unclear whether the reported model performance reflects genuine wildfire lifecycle understanding or mainly the ability to reproduce the authors' labeling policy. This is particularly important because several lifecycle states are defined using manually designed thresholds and temporal transition rules.
  2. The difficulty of Task B is not sufficiently established against simple persistence-based forecasting. Although the paper includes a Historical Average baseline, it does not report a last-value or persistence baseline for next-day personnel, nor persistence/recent-trend baselines for daily cost. Since personnel demand and cost increments may be highly autocorrelated, these missing baselines make it difficult to determine how much of the reported forecasting performance reflects meaningful temporal modeling beyond short-term persistence.
Questions for authors
  1. What fraction of the automatically generated Task A lifecycle labels are modified, deferred, or discarded during expert review? What are the inter-annotator agreement and Cohen's κ before adjudication, both overall and for each lifecycle phase?**
  2. Can the authors report the performance of simple persistence baselines for Task B, particularly for next-day personnel and daily cost? How much improvement do the reported models provide over these simple baselines?**
  3. How were the numerical thresholds used to construct the five lifecycle states selected? Are these thresholds supported by established wildfire-response guidelines or prior literature, and how sensitive are the resulting labels to reasonable changes in these thresholds?**
Reviewer_ou6rOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Existing wildfire benchmarks summarize incidents but miss daily changes, stage transitions, and next-day needs. FireResBench includes 33,303 fire-day records from 5,728 incidents across the contiguous USA (2017–2020). Each record covers one incident on one date and supports stage identification and next-day personnel and cost prediction. We built the dataset as follows:

  1. Start with public wildfire data ICS-209-PLUS and combine it with satellite data of active fires from NASA FIRMS.
  2. Reports are organized chronologically for each incident. Same day (multiple reports), retain the latest valid observation. Missing dates unfilled.
  3. Use an automated pipeline to assign one of five lifecycle labels (like initial attack, rapid escalation, extended attack, containment, or mop-up/monitoring. The rules consider fire age, growth, containment, satellite activity, personnel, and costs.)

The authors perform four main experimental analyses based on FireResBench.

  1. The authors test eight methods using an 80:10:10 data split, keeping each incident in one split. FT-Transformer performs best on the main metrics.
  2. Models often mix up nearby fire stages and make larger prediction errors when more personnel are needed.
  3. Fire conditions help most with identifying stages, while past personnel and costs help most with next-day predictions.
Critical strengths

SC1) A good dataset for wildfire response modeling: The dataset tracks daily changes in fire conditions, stages, personnel, and costs across 5,728 wildfires.

SC2) Careful construction of prediction tasks: The tasks use only current and past data to predict the next day, keeping each wildfire in one split to avoid data leakage.

Critical weaknesses

CW1) The paper mentions expert review but does not report how many labels were checked or corrected, or how often reviewers agreed. This makes it hard to judge the reliability of the lifecycle labels used in Task A.

CW2) It reports prediction scores but does not show how the results help wildfire planning or why the benchmark matters to the wider computing community. Planning how the ICLR or cs community cares about this benchmark.

Questions for authors

W&Q. What does the Stage I rule program score on its own?

The models use the same inputs as the rules that created the labels. Could the authors add the Stage I rules alone to Table 3 to show whether the models do better?

W2&Q2. How much did human review actually change, and who did it?

The paper mentions human review but gives no numbers. How many reviewers took part, how many fire-days were checked, corrected, or excluded, and how often did reviewers agree before resolving disagreements?

W3&Q3) How many samples does Task B really have?

Table 6 lists all 33,303 fire-days for Task B, but only days with a next-day record qualify. How many personnel and cost samples remain, and how many were dropped because reported costs decreased?

W4&Q4) Personnel changes little from day to day. How well does using today’s personnel and the previous day’s cost increase predict tomorrow, and what does HA average?

W5&Q5. Table 3 and Appendix B.2 report conflicting transition accuracies for XGBoost and FT-Transformer. Which is correct, or do they use different data, runs, or definitions of a transition day?

Two smaller points also need clarification:

a. The AI use statement names FireAgentBench. Is this a typo or a different dataset?

b. Why is the 99th percentile for next-day personnel missing from Table 2?

Reviewer_SpaPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a dataset containing U.S. wildfire incident reports (ICS-209-PLUS) and NASA FIRMS satellite fire detections into a per day record. Every data record describes a single fire incident tracked through several fire days (calendar dates; days without a report are left as gaps). Every day’s record for the fire provides information such as area and growth, containment, reported fire-behavior flags, personnel and cumulative cost and related aggregate satellite fire detections. Thereafter they provide labels for two separate tasks: classification and forecasting. The classification task helps identify the stage of the fire — initial attack, rapid escalation, extended attack, containment, mop-up. The second task’s labels provide information about next-day personnel assigned and daily cost. They evaluate different baselines trained on this dataset. The goal of these baselines is to support incident-level situational awareness and next-day aggregate resource demand. The dataset is not merged with typical weather, fuel, spatial topography that is commonly utilized in these kinds of datasets.

Critical strengths
  • It provides aggregate cost and resources required per fire incident

  • it provides a joined dataset with satellite fire detection.

Critical weaknesses
  • There are some basic weaknesses in the paper. First, t is not clear how good the training labels are. The paper never shows why the lifecycle labels should be trusted: there is no count of annotators, no number of Fire-Days reviewed or corrected.

  • It is fundamentally not clear what the value is of learning or forecasting aggregate information that is already present in the record. Fire propagation and response are spatio-temporal processes, commonly modeled as sequential decision problems (MDP/POMDP formulations), and propagation modeling used for response operates at much finer spatial and temporal scales than a day. The paper does not analyze what is lost by projecting this onto daily, incident-level aggregates with no weather, fuel, topography or fine-grained grid location.

  • It is fundamentally not clear what is the value of learning or forecasting aggregate information that is present in the record for this work. While it may be useful for forensic analysis and scenario modeling, it will be very difficult to be used in real-time cases. A good paper for to be considered for comparison is a neurips paper called WildfireDB. Also it will be good to consider the NSF Fire program (https://www.nsf.gov/funding/opportunities/fire-fire-science-innovations-through-research-education) that has an active solicitation and a description of current challenges.

Questions for authors
  • Can you report how the Task A labels were validated — the number of scholars, the number of Fire-Days reviewed and corrected, and the agreement statistics (raw agreement, Cohen’s κ) that Appendix A says are reported?

  • Under what license will the dataset and labels be released, and what are the terms of use of the ICS-209-PLUS and FIRMS sources it is derived from?

  • Can you explain the real-world use case for these forecasts at daily, incident-level granularity, and how they would have been useful in a recent wildfire (for example, the 2025 Los Angeles fires cited in your introduction)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.