- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how evaluation choices, including metrics, matching rules, evaluation scope, and head-selection criteria, affect the reported performance and ranking of wildfire prediction models. The authors formalize these choices as a Fixed Evaluation Contract and introduce the Contract Dominance Ratio (CDR) and Rank Instability Rate (RIR) to quantify score variation and ranking reversals across evaluation configurations. The paper also introduces WILDFIRE-FM, a wildfire-specific reference model, and compares it with several Earth foundation models under multiple evaluation settings. The experiments show that changing the evaluation configuration can substantially change absolute scores and, for some model pairs, their relative rankings. WILDFIRE-FM performs strongly relative to the evaluated models across several reported settings.
Critical strengths
- The paper provides useful empirical evidence of evaluation sensitivity.
- The paper evaluates multiple models under several explicitly reported settings.
Critical weaknesses
-
The methodological novelty and practical role of the proposed evaluation framework are unclear. Keeping the dataset, metric, matching rule, evaluation scope, and model-selection procedure identical is already a basic requirement for fair controlled comparison. A Fixed Evaluation Contract standardizes comparison under one selected setting, but it does not resolve evaluation sensitivity. CDR and RIR quantify score variation and ranking instability, but they do not determine which evaluation setting is operationally appropriate or explain how conflicting conclusions across different reasonable settings should be interpreted.
-
The role of WILDFIRE-FM as a general reference model is insufficiently justified and weakly connected to the main evaluation-sensitivity problem. Different metrics and matching ranges can represent different prediction objectives, such as precise fire-location prediction or broader regional prediction. The paper does not establish that a general WILDFIRE-FM is an appropriate reference for models optimized for these individual objectives. Moreover, stable relative ranking across evaluation settings does not demonstrate that the model itself is insensitive to those settings or that it provides adequate absolute performance under each setting. No model or method is proposed to mitigate the sensitivity emphasized in the motivation.
Questions for authors
-
What is the main methodological contribution of the Fixed Evaluation Contract beyond the standard requirement that all models in a fair controlled comparison use identical evaluation settings? How should researchers select an appropriate contract, especially when different reasonable contracts produce conflicting model rankings?
-
What evidence establishes WILDFIRE-FM as an appropriate reference model across different prediction objectives? In particular, how does it compare with models specifically optimized for exact localization, broader regional prediction, or other individual objectives? If WILDFIRE-FM is not intended to mitigate evaluation sensitivity, why is Evaluation Sensitivity presented as the main problem statement and motivation for introducing the model?