- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to empirically audit the performance of TabPFN 3.5 and TabFM 1.0 models for wildfire occurrence forecasting given geo distribution shifts, with a few local labels and different in-context-examples. They compared tabular foundation models with CatBoost, LightGBM, logistic reg., and a physical baseline (based on FFMC) for wildfire occurrence from two regions in Russia and British Columbia (BC) wildfire records.
They studied: (1) geo transfer of zero label, (2) local adaptation with varying label budgets, (3) prediction averaging for 5 different contexts, (4) stability of response to weather perturbation. (5) also they included synthetic, sequential and archived (FFMC) controls for response fidelity direction.
They found no statistically significant benefit of foundation models for the zero label geo transfer. Prediction averaging works for foundation and classical tree ensemble (Cat/LightGBM). There are mixed results for some local adaption with varying label budgets that appear to favor foundation models. However frozen BC 2024 replicate does not show any clear model-specific benefit. They also show the possibility that predictive accuracy, stability of response and fidelity to physical reference equation can differ.
Critical strengths
- Thoughtful evaluation setup and clear reporting of negative findings.
Nice efforts are made to make evaluations in geographically and temporally separated way, matching label budgets, context-averaging (both for foundation models and classical learners), and using paired block-based uncertainty estimation. Notably, they carry out an additional BC 2024 replication with frozen protocol, and report a failure to pass the prespecified confirmation rule.
All these things make the empirical analysis more credible. The clear mention of pending comparisons and failed replication is particularly useful as positive findings on foundation models may be quite sensitive to evaluation choices.
- Interesting disentangling of the predictive ability and the reliability of response to weather.
The work makes an interesting step beyond the traditional predictive measures, by investigating the variation in weather perturbation response across admissible contexts. The synthetic/sequential/archived FFMC controls serve as a useful benchmark to evaluate the fidelity of response direction.
It is also particularly interesting to see that one can get better predictive fit together with more degraded direction category error. The matched coverage comparison also demonstrates that unanimous context agreement may not be a better way to judge the reliability of directions than the response magnitude.
Critical weaknesses
- The main positive base model comparisons do not prove supremacy over the best classical models in the same comparison classes.
As much of the positive BC 2023 comparisons are presented in Section 5.1 and Table 2 and center on the positive results as compared to a logistic regression ensemble (single context validation choice), yet the paper shows 5 context CatBoost and LightGBM models are much better competitors across various label budgets.
TabPFN, TabFM, CatBoost, and LightGBM have nearly the same 5 context recall at 64 label budget, e.g. 11.5-11.6% around. Foundation models have higher point estimates at higher label budgets, but the paper itself explicitly says no direct inferential tests compared to bagged trees for the BC 2023 comparisons.
So much of the significance of the positive advantages is highly dependent on the comparator selection process, so it's difficult to know if the main positive is that positive or if FMs have a real practical advantage when following the recommended evaluation protocol by the authors.
It would go a long way if there are direct paired comparisons shown against 5 context CatBoost and LightGBM (with simultaneous CIs for AP and fixed budget recall etc.).
- Little independent evaluation evidence and unclear chronology of protocol freezing.
Only one jurisdiction targeted in external evaluation and 31 forecast initializations dates per season. While the authors rightly use moving block uncertainty estimation, there is considerable overlap in forecast results over time and few independent initialization dates, so there are statistical power / external generalizability concerns.
It's nice to have the BC 2024 replication, but only 2 of 16 simultaneous intervals show positive difference for FM. The rest is unclear, so it's not clear how substantial and/or reliable these seeming advantages are.
One issue is the ambiguity about when things were evaluated: it seems that the BC protocol was frozen prior to the calculation of aggregate BC metrics (Introduction), whereas the local adaptation experiment appears to have been set up after looking at the zero-label result (Section 3). But if the zero label result involves BC 2023 then it's not clear which things were decided upon before looking at BC 2023 outcomes vs. which things were decided after.
That's important because it affects how you'd view the BC 2023 adaptation outcomes as confirming. (Of course this is somewhat buffered because BC 2024 evaluation is separately frozen, but still not clear what the fine-grained chronology looks like.)
Questions for authors
Q1. Clarification and more analysis regarding direct comparison with ensemble baselines.
Please give direct paired concurrent confidence intervals for comparison of 5 context TabPFN vs TabFM and 5 context CatBoost and LightGBM on BC 2023 and BC 2024 (for integrated AP and 5%-alert recall). Also, what if you assess all the classical model candidates with the same 5 context method before choosing your comparison model?
This is the core of your claims about the benefits of foundation models with local supervision.
Q2. Clarification of the timing of evaluation.
Please give a fine-grained chronology of decisions about the protocol and inspection of outcomes. For example, what is the relationship between "we froze the BC protocol before we did any computation of aggregate BC metrics" and "our local adaptation experiment was set up after inspection of the zero label result"?
What things about BC 2023 adaptation experiment were decided before looking at BC 2023 outcomes?
Q3. Clarification and robustness analysis for statistical uncertainty.
How much independent temporal information is there effectively, with 31 dates of initialization per season and so much overlap in forecasting horizons for the 35 day moving block intervals?
Are the CIs you report also capturing the uncertainty due to different independently-sampled contexts sets, or are they conditioning on the 5 that you sample? Does the main results of BC 2024 (based on block-lengths) remain consistent on the shorter block-length sensitivity analysis?
Q4. How weather-response instability should be interpreted. Further clarification/sensitivity analysis.
Out of the 27-28% of contexts with disagreements at 4096 labels, how much of that corresponds to actually different (non-flat) directions of responses? vs. flat/non-flat transitions?
How much the above disagreement percentages depend on the specific choice of threshold for ±0.05 (log-odds)? To understand the nature of the instability observed (important differences in behavior vs. sensitivity to how you categorise the responses).