acceptodds
Under review as a conference paper at ICLR 2027

An Open National Benchmark for Forecasting Registered Wildfire Occurrence

Abstract

Machine-learning benchmarks for wildfire forecasting are built almost entirely on satellite fire observations, yet where those products have been compared with agency records they detect a minority of the smaller fires those registers hold, and detection improves steeply with fire size. We release an open benchmark for forecasting where forest fires will be registered: 83 regions of Russia, selected by the continuity of their fire records, 64,620 hexagonal cells of about 253 km², daily over the April to October seasons of 2000 to 2024, labelled with 420,995 registered fires; 0.11% of cell-days carry at least one. Every input is openly licensed and redistributable, with weather sampled on a fixed hierarchical grid that keeps every cell within 68 km of its weather, so the dataset can be published in full and served in real time from the same sources. The protocol is built for extreme imbalance: a temporal split, evaluation on the complete national grid rather than on sampled negatives, and metrics a dispatcher can act on, above all the share of the day's fire cells caught inside a fixed alarm budget. Across 807 days of four fire seasons, a gradient-boosted baseline catches 80.5% of the day's fire cells while flagging 15.2% of the grid, against 63.7% for climatology, 53.6% for the Canadian Fire Weather Index and 47.9% for the Russian statutory index at the same budget; it beats climatology on 97% of individual days and leads at every budget we scored, from 1% to 25% of the grid. Capture varies little across fire size, so the model finds small fires as well as large ones. Matched-representation experiments then show that the hand-designed fire-weather summaries this field relies on, including the statutory index, are replaceable: the same model given the raw 30-day weather series in their place matches or beats them, while a recurrent model and a graph over weather channels show no consistent advantage over it.

Primary area:
datasets and benchmarks
Keywords:
wildfire forecasting, benchmark dataset, rare-event verification, extreme class imbalance, spatiotemporal representation learning, open data, gradient boosting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ryy2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission releases the first open national benchmark built on agency-registered occurrence: 83 regions of Russia selected for record continuity, 64,620 H3 hexagonal cells of about 253 km², daily over April–October 2000–2024, 420,995 registered forest fires, and a positive rate of 0.11% of cell-days. Every input is openly licensed and redistributable, so the dataset can be published whole rather than as a download recipe. The protocol is designed for extreme imbalance: a fixed temporal split, evaluation on the complete national grid rather than on sampled negatives, and rare-event verification scores borrowed from operational meteorology.

Critical strengths
  1. The paper examined the source of labels for 10 publicly available wildfire datasets one by one, proving that none of them used official registration records.

  2. Full redistributability is meaningful. The work releases the frozen dataset with checksums and rebuild scripts, rather than the common download-recipe approach.

  3. The representation study uses identical rows, paired seeds, and a 12-variable control; the claim is correctly stated as absence of evidence.

  4. The protocol is useful: Full-grid evaluation at a fixed alarm budget on sampled negatives, imported rare-event verification scores, and a harness with a fixed contract that scores every submission identically.

Critical weaknesses
  1. Labels may measure monitoring, not fire. The labels come from agency records, which only exist where the agency was actively looking. So the model may be learning where fires get reported rather than where fires actually start, and the paper never runs an experiment to separate the two.

  2. Same-day weather is not a forecast. The model can use the weather on the day it is predicting, making it a nowcast rather than a forecast. No experiment shows how much skill is lost once real forecast weather with a 1–7 day lead time is used.

  3. The index comparison is confounded by resolution. Each fire-danger index gets one weather value per 68 km, shared by about 41 cells, and ties are broken at random. That means part of the 26-point gap reflects finer input information rather than a better method.

  4. The baseline is too weak for a benchmark. Only a single LightGBM model is compared, with no published fire-occurrence model, no spatial graph network, and no compute cost reported. A benchmark needs at least one strong, recognizable reference method before others can be ranked against it.

Questions for authors

See the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ryy2

Thank you for the review, and in particular for asking what the labels actually measure, which the paper had not tested directly. We agree that this needed an experiment, as did the forecast setting, the resolution of the index comparison and the strength of the baselines. We ran each on the complete national grid, under the harness rules and at the 15.2% alarm budget of Table 3: all 807 test-season days and 69,607 fire cell-days, with 95% bootstrap intervals over days. Two small corrections first: the protocol evaluates on the complete grid rather than on sampled negatives (Section 4), and the 253 km² in the paper was H3's global average cell area, while our cells average 261 km² (192 to 305 km²); we have corrected it in the revised PDF.

"Labels may measure monitoring, not fire." The register records the protection zone where each fire began, and fires reach the register differently in each zone. In control zones a fire may be entered from satellite observation alone and suppression may be suspended, while ground zones are monitored from the ground. A model that had learned where fires get reported would fail where registration depends least on people nearby, as climatology does. Ours does not. In ground zones (10,772 fire cell-days) the model catches 89.6%, the model refitted without weather 85.1% and climatology 81.9%. In aviation zones (23,731) the figures are 74.4%, 59.9% and 51.9%, and in control zones (6,658) 62.2%, 34.4% and 9.2%. Past registrations predict ground-zone fires well and control-zone fires hardly at all, and in control zones 28 of the model's 62.2 points come from the daily weather.

"Same-day weather is not a forecast." Archived IFS forecasts exist for the 2024 season, and we used them at leads of 1 to 7 days with the model unchanged. Capture is 75.55% with analysis weather and 75.15%, 74.89%, 74.46%, 73.35%, 72.16%, 70.77% and 69.23% at 1 to 7 days. Even at 7 days it stays above the 2024 climatology (56.59%) and both indices, which read same-day weather.

"The index comparison is confounded by resolution." Scored at the indices' resolution, with each cell taking the mean over the about 41 cells sharing its weather point, the model captures 74.05%, against 53.6% for FWI and 47.9% for Nesterov (Table 3), so about 6 of the 27 points come from resolution. The same learner given only the weather, the information the indices use, captures 63.91%.

"The baseline is too weak for a benchmark." We added a classic statistical occurrence model and a spatial graph network, both fitted on the same years as the released model (2000 to 2020). A logistic model in the form of Martell et al. (1987), with the Canadian moisture codes and seasonal terms fitted per region, captures 62.60%; a national spline logistic model that adds the cell's fire history and position captures 70.93%. The spatial graph network, two GraphSAGE layers over the H3 neighbours of every cell reading the released model's 106 features, captures 79.82% over three seeds, 0.51 to 0.75 points below the released model (80.49%) in every seed. The recurrent and channel-graph models of Table 6 capture 80.59% and 79.05% on the full grid. On cost, one head of the released model fits in 3.7 minutes on a 6-core CPU, and scoring the whole grid takes 6 to 7 seconds a day; the neural models train in 10 to 18 minutes each on one consumer GPU.

Appendix C of the revised PDF reports these experiments, including the zone table, and Section 7 now states the zone result. Their inputs, code and result files are in version 2 of the release, at the paper's anonymous link beside version 1. We hope these results address your concerns, and we would be glad to provide any further detail.

Thank you again for your thoughtful review. As the discussion period is coming to a close, we would like to briefly follow up on our earlier response. The new results are described there and reported in Appendix C of the revised PDF. We would be glad to clarify any remaining point. Thank you for your time

Reviewer_YbayOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an open benchmark for predicting registered fires across Russia. It includes 420,995 fires across 64,620 cells in 83 regions. A tree model ranks cells each day under a fixed alarm budget.

The paper also compares weather representations and learning methods. Its main contribution is the dataset and evaluation protocol.

Critical strengths
  1. The dataset addresses a useful prediction target.

The benchmark studies registered events that agencies use when organizing their response. This target differs from satellite fire detections. The paper also explains that registration records can themselves include satellite-based observations. That distinction makes the task definition more credible. [Sections 1–3, pp. 1–3]

The manuscript describes a frozen release with licensing information, fixed splits, code, and evaluation scripts. A frozen snapshot reduces dependence on changing upstream services. These choices could help other researchers build directly on the benchmark. [Section 3.3, p. 4; Table 1, p. 4]

  1. The main evaluation preserves the difficult class balance.

The main evaluation keeps every cell in the national grid. This preserves the large number of no-fire cases. At an alarm budget of 15.2%, the model captures 80.5% of fire cells. The seasonal historical baseline captures 63.7% at the same budget. [Table 3, p. 6]

The paper also reports alarms per captured fire cell and results across other budgets. These measures make the decision tradeoffs easier to judge. Including historical baselines is especially useful because they outperform both fire indices in this setting. [Figure 1 and Section 5, pp. 6–7]

  1. The reporting makes limitations visible.

I appreciate the direct discussion of what the register does and does not record. The authors explain recording changes, mixed observation sources, and regional weaknesses. They also report poor probability calibration instead of relying only on ranking scores. [Sections 3.1 and 5, pp. 3 and 7]

Results by fire size and year provide useful checks. Capture reaches 79.8% for both 1–10 hectare fires and fires of at least 1,000 hectares. This supports the benchmark’s relevance to smaller registered fires. The yearly results also show that performance varies across seasons. [Table 4, p. 7; Table 7, p. 14]

Critical weaknesses
  1. The information available before prediction needs checking.

The evaluation uses weather for the prediction day. It does not measure performance using a fixed forecast issued before that day. This retrospective task remains useful, but its value for advance warnings remains uncertain. [Section 4, p. 5; Section 7, p. 9]

The 2023 land-cover composite could also contain burns from the 2023 test season. The authors acknowledge this possibility. They have not reported the proposed rerun using the 2020 composite. [Section 4, p. 5]

I do not regard leakage as established. However, weaker performance in 2023 cannot rule it out. This unfinished check is my main concern because it affects which inputs future benchmark users may safely use.

  1. Changes in the data complicate interpretation.

The weather product and fire recording pattern change around 2017. The defined training split ends in 2017, although the released baseline also fits the validation years through 2020. It therefore sees both periods. [Sections 3.1 and 3.3, pp. 3–4; Section 7, p. 9]

These changes do not automatically invalidate the benchmark. However, the paper does not isolate how they affect its main results. A fixed-product weather test would address one source of uncertainty. A sensitivity test within the later recording regime would address the other.

The annual results are helpful, but they do not separate weather-source changes from changes in registration. The paper states that a download using one fixed weather product remains in progress. Completing that check would make the benchmark easier to interpret. [Section 7, p. 9]

  1. Several comparisons support limited practical conclusions.

About 41 cells share each weather point. The fire indices therefore cannot express the same spatial detail as the learned model. The model also receives land cover, population, and fire history. This comparison combines differences in methods, spatial resolution, and information. The authors correctly recognize this limit. [Sections 5 and 7, pp. 6 and 9]

The representation study provides a separate limitation. It uses sampled no-fire cases and three paired seeds. It does not compare each representation’s capture over the full national grid. The restricted-variable control helps address unequal weather inputs, but the remaining differences are small. [Table 6 and Section 6, pp. 8–9]

I would like the representation comparison at the main alarm budgets. A region-calibrated or resolution-matched index baseline would also clarify the practical advantage. These are secondary concerns about the strength of the conclusions.

Questions for authors

How do the 2023 and overall results change when you replace 2023 land cover with the 2020 map? What performance remains using weather forecasts available before the prediction day? Do the main gains remain with one weather product and training within the later recording regime? Can you compare weather representations on the full grid and give the index baselines comparable spatial information?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ybay

Thank you for the careful and detailed review. We agree that the 2023 land-cover check had to be completed, and that the forecast setting, the 2017 changes, the representation study and the index baselines each needed a direct test. We ran all of them on the complete national grid, under the harness rules and at the 15.2% alarm budget of Table 3: all 807 test-season days and 69,607 fire cell-days, with 95% bootstrap intervals over days and differences paired on the same days. Our evaluation code first reproduces the baseline of Table 3 (80.49% [79.1, 81.8]).

Land cover of 2023. We rescored every 2023 test day with the 2020 land-cover and land-cover-change layers, keeping the model unchanged. 2023 capture moves from 76.06% to 76.15% (14,351 instead of 14,334 of 18,846 fire cell-days), and capture over all days from 80.49% to 80.51% (+0.02 points [-0.01, +0.06]). Giving 2024 the 2020 layers as well yields 80.54%. Removing the post-2020 composites never lowers capture, so we find no sign that the 2023 composite encodes that season's burns.

Forecasts available before the day. The provider archives IFS forecasts (0.25 degree) from 2024 onward. For the 2024 season we built each day's inputs from the run issued L days earlier, keeping the analysis up to the issue date. With the model unchanged, capture is 75.55% with analysis weather, and 75.15%, 74.89%, 74.46%, 73.35%, 72.16%, 70.77% and 69.23% at leads of 1 to 7 days. Even at 7 days it stays above the 2024 climatology (56.59%), the model without weather (60.12%), and FWI (47.63%) and Nesterov (42.43%) on same-day weather. The forecasts carry real information: a model refitted on weather known 3 or 7 days earlier reaches only 67.50% and 63.73% on the same days.

One weather product and the later registration regime. We rebuilt the dataset on ERA5 and ERA5-Land requested with the model pinned, retrained the model and recomputed both indices on the same weather. The model captures 80.76% [79.4, 82.0], against 53.67% for FWI and 47.61% for Nesterov. Trained only on 2018-2020, inside the later registration regime, it captures 79.44%, against 62.63% for a climatology built from the same three years; the lead over climatology is 16.8 points, as in Table 3. With both controls together, capture is 79.68%.

Representations on the full grid. We refitted the four models of Table 6 (right pair) at all three seeds, reproducing their panel scores, and scored them on all 807 days. The three-seed means at the main budget are 81.35% for the raw-window tree, 80.59% for the recurrent model, 80.33% for the designed summaries and 79.05% for the channel graph. The raw window leads the designed summaries in each of the three seeds, by 0.99 to 1.08 points, with each seed's interval excluding zero. The lead is 0.7 to 1.0 points from the 5% budget upward and 0.1 to 0.2 points at 1-2%, and among the top 100 cells the designed summaries are marginally ahead (5.5% against 5.3%). The recurrent model and the channel graph trail the raw window in every seed, by 0.76 and 2.30 points on average, so neither shows an advantage over it on the full grid either.

Index baselines with comparable spatial information. Scored at the indices' resolution, with each cell taking the mean over the about 41 cells that share its weather point, the model captures 74.05%, so about 6 of its 27 points over FWI come from resolution. Calibrated to each region's registered fire rate by month and index level, fitted on 2000-2020, FWI reaches 62.68% and Nesterov 57.37%.

In the revised PDF, Appendix C reports each of these tests, Sections 4 to 7 now state them where the original text had left them open, and the mean cell area is corrected to 261 km². Their inputs, code and result files are in version 2 of the release, at the paper's anonymous link beside version 1. We hope these results address your concerns, and we would be glad to provide any further detail.

Thank you again for your careful review. As the discussion period is coming to a close, we would like to briefly follow up on our earlier response. The new results are described there and reported in Appendix C of the revised PDF. We would be glad to clarify any remaining point. Thank you for your time

Overall, I find that the authors have satisfactorily addressed most of my concerns. The rebuttal significantly improves the credibility of the empirical findings, and I am inclined to revise my assessment more positively, provided that the additional experiments and corresponding limitations are clearly incorporated into the final manuscript.

Reviewer_Si4wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an open national benchmark for predicting registered wildfire occurrence in Russia. It covers 83 regions and 64,620 hexagonal cells over the April–October fire seasons from 2000 to 2024, with the final season observed through 12 September. Labels derive from 420,995 agency-registered fires rather than satellite-derived occurrence products. The release includes openly licensed inputs, fixed temporal splits, baseline models, and an evaluation harness.

The main evaluation measures capture of registered fire cells under a fixed daily alarm budget on the complete national grid. At a budget of approximately 15.2%, the gradient-boosted baseline captures 80.5% of fire cells, compared with 63.7% for seasonal climatology, 53.6% for the Canadian Fire Weather Index, and 47.9% for the Russian statutory index. Additional analyses examine performance across fire sizes, seasons, and regions, and compare raw weather sequences with designed fire-weather summaries. The reported results use analysis-day weather and therefore establish same-day risk estimation rather than forecasting skill at a specified lead time.

Critical strengths

A useful benchmark contribution based on administrative fire records. Registered occurrences provide a complementary target to satellite observations, with particular relevance to the events recorded by fire-management agencies. The national scale, long temporal coverage, natural class imbalance, and redistributable inputs make this a potentially valuable resource for rare-event prediction and distribution-shift research. The principal contribution is the dataset and evaluation protocol rather than a new forecasting method.

An informative evaluation protocol with transparent reporting. Evaluating on the complete grid avoids the distorted operational interpretation that can arise from sampled negatives. Capture under a fixed daily alarm budget gives the results a clear decision-making interpretation, while climatology provides an important competitive baseline. Analyses across fire sizes, seasons, and regions reveal variation hidden by the aggregate score. The paper also carefully distinguishes administrative labels from complete ground truth and ranking performance from probability calibration.

Critical weaknesses

The temporal evaluation crosses changes in both weather provenance and registration practices. Both changes occur around 2017, near the original training–validation boundary. Although the released baseline is fitted through 2020 and therefore observes both regimes, the evaluation does not isolate generalization within a stable data-production and registration regime. This limits interpretation of the benchmark’s temporal generalization results. The planned pinned-weather re-extraction and an evaluation restricted to the post-2018 registration regime would materially strengthen the evidence.

The forecasting framing and index comparisons exceed what the current experiments isolate. Analysis-day weather supports same-day risk estimation, but performance with forecast weather at specified lead times remains unmeasured. Moreover, the learned model receives cell-level terrain, land cover, population, and fire-history information, whereas the operational indices share values across approximately 41 cells per weather point. Their performance gap therefore combines differences in information access, spatial resolution, and modelling approach. These limitations do not invalidate the benchmark, but they constrain the forecasting and comparative conclusions.

Questions for authors
  1. Evaluation across data regimes: Can the authors report results using a single pinned weather product and a temporal evaluation restricted to the post-2018 registration regime? How much do the main capture results change under these controls?

  2. Forecasting information set: At what issuance time would the analysis-day weather variables be available? Can the authors provide a fixed-lead-time experiment using archived forecasts, even for the available portion of 2024? If not, would they make the same-day nature of the task more prominent in the abstract and discussion of the title and main results?

  3. Fairness of the index comparison: How much of the advantage remains when the learned baseline is evaluated at the indices’ weather-grid resolution and with a comparable information set? This would help distinguish the value of additional spatial information from differences in modelling quality.

  4. Potential within-season information: Has the proposed re-scoring of 2023 using the 2020 land-cover composite been completed, and does it materially change the headline results?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Si4w

Thank you for a constructive review and for the precise questions. We agree that the evaluation had to be separated from the 2017 changes, that the forecast setting needed a direct test, and that the index comparison mixes resolution, information and modelling. We ran each experiment you asked for on the complete national grid, under the harness rules and at the 15.2% alarm budget of Table 3: all 807 test-season days and 69,607 fire cell-days, with 95% bootstrap intervals over days and differences paired on the same days. Our evaluation code first reproduces the baseline of Table 3 (80.49% [79.1, 81.8]).

Evaluation across data regimes. We rebuilt the dataset on ERA5 and ERA5-Land requested with the model pinned, retrained the model and recomputed both indices on the same weather. The model captures 80.76% [79.4, 82.0], 0.27 points above the paper's model, against 53.67% for FWI and 47.61% for Nesterov. Trained on 2018-2020 only, it captures 79.44%, against 62.63% for a climatology built from the same years. With both controls together it captures 79.68%. Each control moves the main result, and its lead over each baseline, by about one point or less.

Forecasting information set. The analysis-day variables aggregate the 24 UTC hours of the target day, so they are complete only after that day ends; the paper's numbers are a same-day estimate, as you say. For the available part of 2024 we used the provider's archive of IFS forecasts (0.25 degree), building each day's inputs from the run issued 1 to 7 days earlier. Forecast snow is not archived, so we kept the snow analysed at issue. With the model unchanged, capture is 75.55% with analysis weather and 75.15%, 74.46%, 72.16% and 69.23% at leads of 1, 3, 5 and 7 days: a loss of 0.41 points [0.04, 0.80] at 1 day and 6.32 points [5.07, 7.68] at 7. Over all four years, a model refitted on the weather known 1, 3 or 7 days before reaches 78.85%, 74.39% and 71.89%, against 63.73% for climatology.

Fairness of the index comparison. Scored at the indices' resolution, with each cell taking the mean over the about 41 cells sharing its weather point and ties broken at random as for the indices, the model captures 74.05%, so about 6 of its 27 points over FWI come from resolution. With a comparable information set, the same learner given only the 25 weather features, whose scores are shared by all cells of a weather point as the indices' values are, captures 63.91%, against 53.6% for FWI and 47.9% for Nesterov (Table 3): with both controls, about 10 of the 27 points remain. Calibrated to each region's registered fire rate by month and index level, FWI reaches 62.68% and Nesterov 57.37%.

Potential within-season information. The rescore is done. With the 2020 land-cover layers, 2023 capture is 76.15% against 76.06%, and 80.51% against 80.49% over all days; giving 2024 the 2020 layers as well yields 80.54%. We find no sign of within-season information.

In the revised PDF, the abstract and Section 4 now state the same-day information set and the forecast results, Section 5.1 reports the recalibrated index, Section 7 reports the pinned-weather and post-2018 results, and Appendix C lists every new control. Their inputs, code and result files are in version 2 of the release, at the paper's anonymous link beside version 1. We hope these results address your concerns, and we would be glad to provide any further detail.

Thank you again for your constructive review. As the discussion period is coming to a close, we would like to briefly follow up on our earlier response. The new results are described there and reported in Appendix C of the revised PDF. We would be glad to clarify any remaining point. Thank you for your time

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.