- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submission releases the first open national benchmark built on agency-registered occurrence: 83 regions of Russia selected for record continuity, 64,620 H3 hexagonal cells of about 253 km², daily over April–October 2000–2024, 420,995 registered forest fires, and a positive rate of 0.11% of cell-days. Every input is openly licensed and redistributable, so the dataset can be published whole rather than as a download recipe. The protocol is designed for extreme imbalance: a fixed temporal split, evaluation on the complete national grid rather than on sampled negatives, and rare-event verification scores borrowed from operational meteorology.
Critical strengths
-
The paper examined the source of labels for 10 publicly available wildfire datasets one by one, proving that none of them used official registration records.
-
Full redistributability is meaningful. The work releases the frozen dataset with checksums and rebuild scripts, rather than the common download-recipe approach.
-
The representation study uses identical rows, paired seeds, and a 12-variable control; the claim is correctly stated as absence of evidence.
-
The protocol is useful: Full-grid evaluation at a fixed alarm budget on sampled negatives, imported rare-event verification scores, and a harness with a fixed contract that scores every submission identically.
Critical weaknesses
-
Labels may measure monitoring, not fire. The labels come from agency records, which only exist where the agency was actively looking. So the model may be learning where fires get reported rather than where fires actually start, and the paper never runs an experiment to separate the two.
-
Same-day weather is not a forecast. The model can use the weather on the day it is predicting, making it a nowcast rather than a forecast. No experiment shows how much skill is lost once real forecast weather with a 1–7 day lead time is used.
-
The index comparison is confounded by resolution. Each fire-danger index gets one weather value per 68 km, shared by about 41 cells, and ties are broken at random. That means part of the 26-point gap reflects finer input information rather than a better method.
-
The baseline is too weak for a benchmark. Only a single LightGBM model is compared, with no published fire-occurrence model, no spatial graph network, and no compute cost reported. A benchmark needs at least one strong, recognizable reference method before others can be ranked against it.
Questions for authors
See the weakness section.
2 comments
Thank you again for your thoughtful review. As the discussion period is coming to a close, we would like to briefly follow up on our earlier response. The new results are described there and reported in Appendix C of the revised PDF. We would be glad to clarify any remaining point. Thank you for your time
Response to Reviewer Ryy2
Thank you for the review, and in particular for asking what the labels actually measure, which the paper had not tested directly. We agree that this needed an experiment, as did the forecast setting, the resolution of the index comparison and the strength of the baselines. We ran each on the complete national grid, under the harness rules and at the 15.2% alarm budget of Table 3: all 807 test-season days and 69,607 fire cell-days, with 95% bootstrap intervals over days. Two small corrections first: the protocol evaluates on the complete grid rather than on sampled negatives (Section 4), and the 253 km² in the paper was H3's global average cell area, while our cells average 261 km² (192 to 305 km²); we have corrected it in the revised PDF.
"Labels may measure monitoring, not fire." The register records the protection zone where each fire began, and fires reach the register differently in each zone. In control zones a fire may be entered from satellite observation alone and suppression may be suspended, while ground zones are monitored from the ground. A model that had learned where fires get reported would fail where registration depends least on people nearby, as climatology does. Ours does not. In ground zones (10,772 fire cell-days) the model catches 89.6%, the model refitted without weather 85.1% and climatology 81.9%. In aviation zones (23,731) the figures are 74.4%, 59.9% and 51.9%, and in control zones (6,658) 62.2%, 34.4% and 9.2%. Past registrations predict ground-zone fires well and control-zone fires hardly at all, and in control zones 28 of the model's 62.2 points come from the daily weather.
"Same-day weather is not a forecast." Archived IFS forecasts exist for the 2024 season, and we used them at leads of 1 to 7 days with the model unchanged. Capture is 75.55% with analysis weather and 75.15%, 74.89%, 74.46%, 73.35%, 72.16%, 70.77% and 69.23% at 1 to 7 days. Even at 7 days it stays above the 2024 climatology (56.59%) and both indices, which read same-day weather.
"The index comparison is confounded by resolution." Scored at the indices' resolution, with each cell taking the mean over the about 41 cells sharing its weather point, the model captures 74.05%, against 53.6% for FWI and 47.9% for Nesterov (Table 3), so about 6 of the 27 points come from resolution. The same learner given only the weather, the information the indices use, captures 63.91%.
"The baseline is too weak for a benchmark." We added a classic statistical occurrence model and a spatial graph network, both fitted on the same years as the released model (2000 to 2020). A logistic model in the form of Martell et al. (1987), with the Canadian moisture codes and seasonal terms fitted per region, captures 62.60%; a national spline logistic model that adds the cell's fire history and position captures 70.93%. The spatial graph network, two GraphSAGE layers over the H3 neighbours of every cell reading the released model's 106 features, captures 79.82% over three seeds, 0.51 to 0.75 points below the released model (80.49%) in every seed. The recurrent and channel-graph models of Table 6 capture 80.59% and 79.05% on the full grid. On cost, one head of the released model fits in 3.7 minutes on a 6-core CPU, and scoring the whole grid takes 6 to 7 seconds a day; the neural models train in 10 to 18 minutes each on one consumer GPU.
Appendix C of the revised PDF reports these experiments, including the zone table, and Section 7 now states the zone result. Their inputs, code and result files are in version 2 of the release, at the paper's anonymous link beside version 1. We hope these results address your concerns, and we would be glad to provide any further detail.