acceptodds
Under review as a conference paper at ICLR 2027

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

Abstract

Earth foundation models can be transferred to wildfire prediction, but their reported downstream scores depend on the evaluation choices used to compute them. We formulate wildfire transfer as a fixed-contract evaluation problem and test how evaluation choices affect scores and model rankings. Holding predictions fixed, decision changes from to solely with the matching rule; holding frozen features fixed, changing the head-selection metric changes decision by up to percentage points. We introduce the contract dominance ratio and rank instability rate, showing that evaluation choices affect scores more than backbone choice and reverse the relative ranking of up to 46% of model pairs across configurations. We develop WILDFIRE-FM to provide reference performance across six downstream tasks under specified contracts. Under matched contracts, this wildfire reference leads the evaluated frozen backbones on occupancy, spread, and burned-area prediction. The reference provides a baseline for comparing Earth-FM transfer across tasks, while the controlled checks show why these comparisons require explicit evaluation contracts. See implementation details at: https://anonymous.4open.science/r/Wildfire-fm-evaluation-contracts-5AE9.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Wildfire Prediction, Earth Foundation Models, Evaluation Contracts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hHUuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how evaluation choices, including metrics, matching rules, evaluation scope, and head-selection criteria, affect the reported performance and ranking of wildfire prediction models. The authors formalize these choices as a Fixed Evaluation Contract and introduce the Contract Dominance Ratio (CDR) and Rank Instability Rate (RIR) to quantify score variation and ranking reversals across evaluation configurations. The paper also introduces WILDFIRE-FM, a wildfire-specific reference model, and compares it with several Earth foundation models under multiple evaluation settings. The experiments show that changing the evaluation configuration can substantially change absolute scores and, for some model pairs, their relative rankings. WILDFIRE-FM performs strongly relative to the evaluated models across several reported settings.

Critical strengths
  • The paper provides useful empirical evidence of evaluation sensitivity.
  • The paper evaluates multiple models under several explicitly reported settings.
Critical weaknesses
  • The methodological novelty and practical role of the proposed evaluation framework are unclear. Keeping the dataset, metric, matching rule, evaluation scope, and model-selection procedure identical is already a basic requirement for fair controlled comparison. A Fixed Evaluation Contract standardizes comparison under one selected setting, but it does not resolve evaluation sensitivity. CDR and RIR quantify score variation and ranking instability, but they do not determine which evaluation setting is operationally appropriate or explain how conflicting conclusions across different reasonable settings should be interpreted.

  • The role of WILDFIRE-FM as a general reference model is insufficiently justified and weakly connected to the main evaluation-sensitivity problem. Different metrics and matching ranges can represent different prediction objectives, such as precise fire-location prediction or broader regional prediction. The paper does not establish that a general WILDFIRE-FM is an appropriate reference for models optimized for these individual objectives. Moreover, stable relative ranking across evaluation settings does not demonstrate that the model itself is insensitive to those settings or that it provides adequate absolute performance under each setting. No model or method is proposed to mitigate the sensitivity emphasized in the motivation.

Questions for authors
  1. What is the main methodological contribution of the Fixed Evaluation Contract beyond the standard requirement that all models in a fair controlled comparison use identical evaluation settings? How should researchers select an appropriate contract, especially when different reasonable contracts produce conflicting model rankings?

  2. What evidence establishes WILDFIRE-FM as an appropriate reference model across different prediction objectives? In particular, how does it compare with models specifically optimized for exact localization, broader regional prediction, or other individual objectives? If WILDFIRE-FM is not intended to mitigate evaluation sensitivity, why is Evaluation Sensitivity presented as the main problem statement and motivation for introducing the model?

Reviewer_cQtGOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
3
Read review
Paper summary

In this work, the authors claim that while earth foundation models (EarthFMs) can be used for wildfire prediction, their fitness for this purpose is insufficiently well evaluated by current benchmarking practices. The authors question whether state of the art methods evaluate how EarthFMs transfer to downstream tasks are not mistaken in not providing a fixed evaluation contract.

The authors train a wildfire foundation model and propose six downstream tasks on which to evaluate future EarthFMs on against, proposing to use their wildfire FM as a reference. The authors further introduce methods and metrics they propose would make the evaluation of EarthFMs more reliable, by enforcing the fixed contract evaluation, fixed output and fixed feature checks along with contract dominance ratio and rank instability rate metrics.

Critical strengths
  • Wildfire prediction is an important, timely topic.
  • The wildfire foundation model seems to be reasonably fit for purpose, at least under the authors evaluation
Critical weaknesses

Unfortunately, the main contribution of the paper is not the wildfire foundation model provided as a reference. The main thrust of the paper revolves around the differences that matching rules and other evaluation decisions have on model ranking. To combat this the authors propose the CDR contract dominance ratio to attribute performance to either model backbone, configuration selection, or a combination of the two. I think the idea that the evaluation metric has an impact on the evaluation results, such as the massive impact of the matching rule is both obvious and well known. Similarly, given the spatial precision required for EarthFM forecasts (even beyond wildfires), the differences between an exact match, and one allowing for a tolerance are well known in literature with many evaluation criteria explicitly focusing on minimizing for example the double penalty of predicting an event mostly correctly, but with an spatial offset. That the ranking of certain FMs would change under changing the evaluation criteria is also not novel, but rather expected.

The CDR metric, arguably the largest contribution, is insufficiently explained and discussed, both in the main paper and in the appendix. As far as I can understand, this metric evaluates each method with a frozen set of candidate heads (using the same cardinality for heads in all models). In the appendix only 2: linear and shallow heads are discussed, but the description suggests 12 head options are available. Given how the authors formulate CDR, the number of heads is crucial, and changing the cardinality of this set is likely to also change the results, and also the relative ranking.

The experimental evaluation fails to convince - the only lesson learned in my opinion is that exact occupancy predictions are very challenging, to the point where none of the models, including the reference, are fit for purpose. The proposed metrics are also questionable as analog retrieval, smoke and extreme heat show very little variation across all models, with the exception of the reference and Alphaearth. While that should be investigated in more detail to understand why, a metric where all (or almost all) models fail or succeed is not a good metric. For example, for smoke, models either report a result normalized to 0.74-0.75 or 1. That is effectively a binary result.

The paper is also written in a needlessly confusing way (terms used before being introduced - CDR, RIR and so on), RQ1 not being actual questions, or being answered. The authors also use references in a misleading way. For example they state "By comparison, existing geospatial benchmarks improve comparability by standardizing a particular evaluation protocol (Lacoste et al., 2023; Marsocci et al., 2024; Simumba et al., 2025), but they do not quantify whether evaluation configuration or backbone identity contributes more to reported wildfire performance, or how often reasonable configurations reverse pairwise Earth-FM rankings". Two of the three references don't even mention or evaluate wildfire prediction at all.

Questions for authors

Why not include the method for selection of the head as part of the overall system and evaluate the system as a whole?

How is the use of fixed contract analysis better at predicting downstream tasks? Being able to differentiate backbone and configuration influences is useful, but where does your evaluation show how the differentiation is helpful in evaluating/deciding which models perform better?

Why are the selected six tasks particularly useful as predictors? The first three are easy to understand/follow - with occupancy and fire spread being clearly supported by literature. The last three seem to have limited value or differentiation. The only explanation provided in the work is "Final burned area, analog retrieval, smoke PM2.5, and extreme heat are supporting tasks whether the transfer performance observed on the primary fire tasks also appears on other wildfire-related prediction problems." This seems arbitrary - can you explain why these tasks are selected?

What is Figure 5 trying to communicate? The caption and text explanation are insufficient to understand the message being conveyed.

Reviewer_7QpGOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how evaluation choices affect wildfire prediction scores and model rankings. It proposes explicit evaluation protocols, measures sensitivity to those protocols across a large array of models, and introduces Wildfire-FM as a wildfire-specific reference model across six tasks. They show that you can see big differences in performance depending on how it is measured and propose a practical methodology to remedy this.

Critical strengths

This paper addresses an important concern, particularly that improvements reported in papers on Earth Foundation Models may reflect primarily differences in evaluation choices.

Showing that evaluation choices can reverse model rankings makes the findings more consequential than simply showing that scores change, confirming the importance of such a study.

The model they train, WILDFIRE-FM, provides a wildfire-trained reference that helps contextualize the performance of general-purpose Earth foundation models.

Critical weaknesses

I found the terminology “evaluation contract” a bit confusing on first reading. An earlier definition and a concrete example of its five components would make the framework easier to understand at first read.

The main text should describe Wildfire-FM’s architecture and pretraining loss more clearly, since the model is a central contribution.

Referring to Wildfire-FM as “wildfire reference” or “wildfire ref.” makes the presentation harder to follow. Consistently using the model’s name would improve clarity of the paper.

The results therefore leave about which factors explain Wildfire-FM’s advantage.

Questions for authors

1.Is it possible to update some of the terminology in the paper, or define these terms before they are used, to improve readability?

2.Can we move Fig. 6 to the main text, and more clearly define Wildfire-FM, as mentioned in the weakness section?

3.Is the strong performance of the Wildfire-FM mostly because it is a wildfire specific model, rather than being a more general Earth Foundation Model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.