- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents FireResBench, an event-centered benchmark for wildfire lifecycle understanding and operational-response forecasting. The benchmark organizes 2017–2020 wildfire incident records from the CONUS into 33,303 Fire-Day instances covering 5,728 incidents. It defines two tasks: (1) predicting the current wildfire lifecycle state from five operational phases, with labels constructed through rule-based labeling followed by domain-scholar review; and (2) forecasting next-day personnel demand and daily cost from temporally valid incident histories. The paper evaluates a broad set of tree-based, tabular-attention, recurrent, and temporal forecasting models. Beyond aggregate metrics, the experiments analyze lifecycle boundaries, demand tails, incident-level response trajectories, feature contributions, historical-context length, temporal OOD generalization, and missingness robustness. The results show that FT-Transformer is generally the strongest baseline, while errors concentrate around intermediate lifecycle boundaries and high-demand operational regimes. Overall, the benchmark addresses a relatively underexplored operational aspect of wildfire modeling, but the empirical validation of label quality and the difficulty/validity of some forecasting targets could be strengthened.
Critical strengths
- The paper addresses a meaningful and relatively underexplored problem. FireResBench moves beyond conventional wildfire benchmarks focused on detection, spread, or burned area by explicitly modeling operational lifecycle states and response-resource demand. The event-centered Fire-Day representation and strict temporal evidence boundary are sensible design choices for this setting. 2. The evaluation is broader than a standard benchmark comparison. In addition to aggregate performance, the paper investigates lifecycle transitions, demand tails, incident-level trajectories, feature ablations, historical context, temporal OOD generalization, and missingness. These analyses provide useful evidence about where current models succeed and fail.
Critical weaknesses
- The validity of the Task A lifecycle labels is not sufficiently quantified. The paper describes a two-stage labeling procedure combining rule-based labeling and domain-scholar review, but it does not report key annotation-quality statistics, such as inter-annotator agreement, correction/defer rates, or the proportion of automatically assigned labels that were modified during expert review. Therefore, it remains unclear whether the reported model performance reflects genuine wildfire lifecycle understanding or mainly the ability to reproduce the authors' labeling policy. This is particularly important because several lifecycle states are defined using manually designed thresholds and temporal transition rules.
- The difficulty of Task B is not sufficiently established against simple persistence-based forecasting. Although the paper includes a Historical Average baseline, it does not report a last-value or persistence baseline for next-day personnel, nor persistence/recent-trend baselines for daily cost. Since personnel demand and cost increments may be highly autocorrelated, these missing baselines make it difficult to determine how much of the reported forecasting performance reflects meaningful temporal modeling beyond short-term persistence.
Questions for authors
- What fraction of the automatically generated Task A lifecycle labels are modified, deferred, or discarded during expert review? What are the inter-annotator agreement and Cohen's κ before adjudication, both overall and for each lifecycle phase?**
- Can the authors report the performance of simple persistence baselines for Task B, particularly for next-day personnel and daily cost? How much improvement do the reported models provide over these simple baselines?**
- How were the numerical thresholds used to construct the five lifecycle states selected? Are these thresholds supported by established wildfire-response guidelines or prior literature, and how sensitive are the resulting labels to reasonable changes in these thresholds?**