- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PyroAdapt, a Pretrain–Retrieve–Rank framework for wildfire occurrence prediction under severe class imbalance, temporal distribution shift, and spatial heterogeneity. A risk model is first pretrained on historical data. For a target period, the unlabeled covariates of the whole target period are used to retrieve similar labeled historical examples, and the pretrained model is fine-tuned on positive–negative pairs from these examples through pairwise ranking losses, without any target labels. The authors compare three ranking objectives, direct ranking, RDPO, and selective ranking, within a unified score-gap formulation, and provide theoretical analysis of their gradient behavior and of a finite-sample bound on missed fire locations under a fixed daily budget. Experiments cover temporal adaptation in Yosemite and location prioritization across California.
Critical strengths
- The problem setting is practically relevant, and the same-day pairing fits it well. Prioritizing the most fire-prone locations under a fixed daily budget matches how monitoring capacity is actually used, and restricting pairs to the same day trains the model to rank locations within a date rather than dates against each other.
Critical weaknesses
- The gains are hard to attribute to retrieval and adaptation. The main comparison is continued focal fine-tuning with a pointwise loss against same-day pairwise ranking. Since the evaluation metrics also measure within-day ranking, the improvement may come from aligning the training objective with the metrics rather than from retrieval or the adaptation framework. Useful controls are missing, including the same ranking objective trained on the full historical data, ranking adaptation without retrieval, and a standard learning-to-rank baseline. It is also unclear whether XGBoost uses the same ecoregion and history features as the proposed model.
- The advantage of selective ranking over direct ranking and RDPO is not established. The differences are small relative to seed variability, and the gradient analysis suggests the three objectives behave similarly under the chosen temperature. Yet the abstract and contributions highlight the additional captures attributed to selective ranking, without confidence intervals or significance tests. The claims about selective ranking should therefore be qualified.
- The higher recall for extreme fires may reflect threshold differences rather than better discrimination. Each policy uses its own F1-optimal threshold, so alarm rates are not comparable. Recall increases while F1 remains largely unchanged, which is consistent with a shifted operating point. The abstract’s claim that the method detects more extreme fire events is therefore not fully supported. Recall at matched alarm rates would show whether the method finds more extreme fires at a comparable alert burden.
- The label is not equivalent to wildfire occurrence. It is a binary indicator derived from burned dry matter after substantial spatial smoothing, which raises the positive rate and makes neighboring locations spatially correlated. The number of captured positives may therefore not correspond to the number of distinct fire events. Evaluation using unsmoothed labels or event-level wildfire records is needed to determine whether the reported gains translate into better detection of actual fire events.
- The retrieval strategy relies on an untested coverage assumption. Nearest-neighbor retrieval is useful only if the archive contains samples similar to the target year’s conditions. The paper acknowledges this limitation but does not evaluate cases where target covariates fall outside historical coverage. The choice k = 5 is fixed without a sensitivity analysis, and a fixed k does not ensure the neighbors are representative. The retrieval ablation uses continued focal fine-tuning rather than the proposed ranking losses, so it does not isolate retrieval’s contribution to ranking adaptation. Retrieval also uses covariates from the entire target year, which raises a practical concern if forecasts are needed before those covariates are available.
- The theoretical results do not fully support the emphasis placed on them. Some results appear to follow from standard arguments, including the reduction of binary-label RDPO. The budget bound also applies to any nonnegative target, rather than specifically to selective ranking, and provides no guarantee for a future year. Calling it a “certificate” may therefore overstate what it establishes. The paper should clarify how its contribution differs from prior work on binary-label RDPO, particularly Jiang and Sun (2026).
- The experiments do not isolate temporal shift or spatial heterogeneity. Yosemite is a single study site, and the California evaluation covers only one test year on the same grid cells. There is no ablation of the spatial inputs and no evaluation on held-out regions. The experiments therefore do not establish whether the method generalizes to locations with different spatial characteristics.
- Several experimental choices need clarification. The ranking temperature differs between the two datasets, but the rationale is not stated. The small number of seeds captures only optimization variability. The paper should report how the temperatures were selected and whether the baselines received comparable hyperparameter tuning.
Questions for authors
Does the gain over continued focal fine-tuning persist when the same ranking objective is trained on the full historical data or on a random subset of the same size, rather than on retrieved samples? These comparisons would help determine whether retrieval provides additional benefits or whether the gains mainly come from aligning the training objective with the evaluation metrics.
6 comments
Response to Reviewer pnHZ (2/3)
W3: high-DM recall at matched alarm rates. We re-evaluated recall within the retrospective top-DM strata at matched budgets. When continued focal raises the same total number of alarms, budget-weighted selective ranking still recalls +27.6 [+5.0, +50.2] pp more of the top-5% DM cell–days, +18.2 [−0.8, +37.2] of the top 10%, and +13.5 [+5.8, +21.2] of the top 20%. The unweighted objectives give +31.5 to +33.3 pp on the top 5%. At 34 cells per day the gains remain positive on average (+12.4 / +10.5 / +8.0 pp), and the intervals mostly include zero. The revision replaces the unmatched figure with this comparison and narrows the abstract to "higher recall of the highest-dry-matter cell–days at an equal alarm budget."
W4: smoothed labels vs. distinct fire events. We re-scored the saved 2022 logits against two outcomes no model was trained on:
- unsmoothed cell-level labels: 2,193 raw-DM-positive cell–days;
- 1,266 distinct fire events, formed by linking raw-positive cell–days in the same or an adjacent cell on the same or next date.
| California 2022, 3 seeds | Raw-label daily AP | Raw-label R@5% | Events detected at 34 cells/day |
|---|---|---|---|
| Continued focal | 6.73 ± 0.56 | 22.73 ± 1.89 | 348 ± 34 |
| Spatial XGBoost | 8.75 ± 0.16 | 29.38 ± 1.24 | 392 ± 2 |
| Direct ranking | 8.48 ± 0.68 | 28.95 ± 4.23 | 400 ± 13 |
| Selective ranking | 8.65 ± 0.44 | 29.71 ± 4.94 | 397 ± 15 |
| Budget-weighted selective | 8.96 ± 0.82 | 30.78 ± 1.87 | 397 ± 16 |
Every ranking objective detects more distinct events than continued focal in every seed under the same daily budget, about 50 more (+14%). The revision adds this table, states in the main text that budget counts are neighborhood-smoothed cell–days, and removes the cell–day count from the abstract.
W5: coverage, k, ranking-loss retrieval ablation, full-year covariates.
- The retrieval ablation now uses the ranking losses (W1 above).
- Sensitivity to k (new). With everything else fixed, budget-weighted selective ranking gives, for k = 1 / 5 / 20 / full archive: 32.09 / 32.48 / 32.00 / 31.73 (validation 2020), 31.94 / 31.96 / 31.27 / 30.99 (validation 2021), and 26.85 / 26.06 / 25.66 / 25.90 (2022) daily AP. On both validation years k = 5 is best or tied with k = 1. Larger neighborhoods are significantly worse in all three years (k = 20 − k = 5: −0.48, −0.69, −0.40). In 2022, k = 1 is higher (+0.79 [+0.38, +1.20]); we keep the pre-specified k = 5, which the validation years support, rather than tuning k on the test year.
- Coverage (new analysis). We split the 2022 dates into tertiles by how far their target covariates lie from the archive (mean distance of each cell to its nearest historical row in the retrieval space). On the third of dates farthest from the archive, budget-weighted selective ranking still gains +2.16 [+0.98, +3.42] daily AP over continued focal, vs. +5.93 [+4.42, +7.50] on the nearest third. Retrieval adds +0.69 [+0.09, +1.32] over the full archive on the middle tertile but nothing on the farthest one (−0.31 [−0.70, +0.10]). This is what the coverage assumption predicts: when target conditions leave the archive, retrieval stops helping, while the ranking gain persists. The revision reports this analysis and states the coverage assumption next to the retrieval step.
- Full-year covariates. The full-archive variant uses no target-period covariates (25.90% vs. 22.35% for continued focal on the same archive), so the operational concern applies to the optional retrieval step only. The revision labels that step as batch-transductive.
Response to Reviewer pnHZ (3/3)
W6: scope of the theory. We agree with the reviewer's characterization.
- The binary-label reduction (Prop. A.1) is a direct algebraic reduction. We keep it only to define the Yosemite binary-label baseline, and the revision labels it as background.
- The daily-budget bound applies to any nonnegative target, and that is its purpose: it identifies which members of the target family control avoidable Top-B misses. Direct and selective ranking do; RDPO, whose targets can be negative, does not. It is a deterministic, in-sample inequality on training dates with no guarantee for a future year. The revision therefore renames it the "Top-B miss bound" and states this scope next to the proposition. The new Corollary 1 extends it to budget-aligned pair weights.
- Relation to Jiang & Sun (2026): that work combines supervised fine-tuning with a DPO preference between the true and the complementary label of each input, for a weather-only Yosemite model. This paper studies input-pair ranking between locations on the same date, a different object whose loss depends on score gaps between candidates and is invariant to per-day score shifts. On top of it, we contribute the score-gap family with its gradient-allocation and sensitivity results (Props. 1–2), the Top-B miss bound and its weighted extension, and the budget-weighted objective. The revision adds this comparison to Related Work.
W7: isolating temporal shift and spatial heterogeneity.
- Spatial-input ablation with the learner held fixed: adding terrain, ecoregion, and causal history raises daily AP from 14.24 to 24.40 for XGBoost and from 14.82 to 21.10 for pretrained TabNet. The ranking gains are measured on top of this context.
- More target years. Beyond 2022, the chronological validation protocol evaluates California 2020 and 2021. There, budget-weighted selective ranking exceeds direct ranking in both years (+0.79 [+0.43, +1.16]; +0.70 [+0.12, +1.33]). Yosemite adds three rolling target years.
- Held-out regions. We agree these are not evaluated. The revision states that the claims concern temporal adaptation on historically observed locations; Appendix E describes a held-out-region design.
W8: temperatures, seeds, tuning parity.
- Temperatures. California's β = 0.1 was a fixed default, never tuned on 2022. On the validation split, direct ranking is flat over β ∈ {0.03, 0.1} (paired difference +0.08 [−2.21, +2.36]) and declines for β ≥ 0.3 (β = 1: −2.64 [−4.75, −0.53]). The default therefore lies on the validation plateau. In 2022, ranking beats pointwise adaptation at every β across a 33× range. Yosemite uses β = 1 for every ranking objective; it is the only value we ran and was not tuned on any Yosemite year.
- Tuning parity. All TabNet objectives share the checkpoint, optimizer, learning rate, batch size, and exact update count. Spatial XGBoost uses fixed tree hyperparameters with no early stopping. Every seeded method is reported the same way, as a three-seed ensemble. We also varied the learning rate, weight decay, and number of updates for the budget-weighted objective on the validation split only. The selected setting differed from the shared default only in weight decay and gave the same 2022 result, so the main table keeps the default shared by all methods (Appendix D.5).
- Seeds. We agree that three seeds capture optimization variability within one year. We therefore report paired intervals throughout, add the 2020–2021 validation years, and keep Yosemite's three rolling target years.
We are grateful for these suggestions; they made the evaluation considerably more rigorous. We would be happy to address any remaining questions.
Response to Reviewer pnHZ (1/3)
We thank the reviewer for a detailed and technically precise review. The suggested controls have substantially sharpened the paper. In the revision, every California comparison uses matched compute (same checkpoint, retrieved set, exactly 6,800 updates) and three-seed ensembles for every method, with paired bootstrap intervals over the 351 fire dates. Design choices are checked on a chronological validation split (pretraining 2002–2019; evaluation 2020–2021) that never touches 2022.
W1 / Question: attributing the gains; full-archive and random-subset controls; standard LTR baseline; XGBoost features.
W2: selective vs. direct ranking and RDPO. We agree. At exactly 6,800 updates on an identical stream of minibatches and pairs, the three targets are indistinguishable (24.40 / 24.39 / 24.39 daily AP; RDPO − direct −0.01 [−0.15, +0.13]). This is the small-margin regime our sensitivity bound (Prop. 2) predicts at β = 0.1, so the reviewer's reading of the gradient analysis is correct. Selective's original Top-5% advantage over RDPO came almost entirely from seed 22 (−0.00 / +1.97 / +0.07). The revision removes the selective-specific capture claim and presents the three targets as comparable members of one family. What distinguishes our method is the budget-aligned weighting, which improves on unweighted selective ranking by +1.16 [+0.72, +1.59] in California 2022. On Yosemite 2021 it improves AUROC (+0.18 [+0.09, +0.27]), AUPRC (+0.25 [+0.05, +0.46]), and R@30 (+1.27 [+0.60, +1.93]), positive in all five seeds.