acceptodds
Under review as a conference paper at ICLR 2027

Auditing Tabular Foundation Models for Wildfire Forecasting: Transfer, Local Adaptation, and Weather-Response Stability

Abstract

Wildfire forecasts must rank rare fire days from weather predictions, often before a region has local fire labels. We evaluate TabPFN and TabFM against CatBoost, LightGBM, logistic regression and the Fine Fuel Moisture Code (FFMC), using complete fire journals from two Russian regions and held-out seasons in British Columbia (BC). Neither foundation model beats its selected classical comparator in any of six zero-label transfer contrasts; FFMC recalls 3–5 percentage points more fires at a 5% alert budget. With 64 local BC labels, averaging five contexts improves both foundation models, but averaged trees reach the same recall. On BC 2023 the foundation models exceed the classical family selected by single-context validation by about 0.9 average-precision points and 1.8–2.0 recall points. The prespecified rule for confirming this advantage on untouched BC 2024 fails: only 2 of 16 simultaneous intervals are positive, although context averaging still helps. Weather-response categories disagree across contexts for 27–28% of queries at 4,096 labels, compared with 17% for logistic regression. Agreement across contexts selects FFMC-teacher response directions no better than response magnitude at matched coverage. Ranking, response stability and FFMC-equation fidelity therefore require separate evaluation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
wildfire forecasting, fire weather, tabular foundation models, in-context learning, context averaging, geographic transfer, explanation stability, physical response fidelity, TabPFN, TabFM.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_87dtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to empirically audit the performance of TabPFN 3.5 and TabFM 1.0 models for wildfire occurrence forecasting given geo distribution shifts, with a few local labels and different in-context-examples. They compared tabular foundation models with CatBoost, LightGBM, logistic reg., and a physical baseline (based on FFMC) for wildfire occurrence from two regions in Russia and British Columbia (BC) wildfire records.

They studied: (1) geo transfer of zero label, (2) local adaptation with varying label budgets, (3) prediction averaging for 5 different contexts, (4) stability of response to weather perturbation. (5) also they included synthetic, sequential and archived (FFMC) controls for response fidelity direction.

They found no statistically significant benefit of foundation models for the zero label geo transfer. Prediction averaging works for foundation and classical tree ensemble (Cat/LightGBM). There are mixed results for some local adaption with varying label budgets that appear to favor foundation models. However frozen BC 2024 replicate does not show any clear model-specific benefit. They also show the possibility that predictive accuracy, stability of response and fidelity to physical reference equation can differ.

Critical strengths
  1. Thoughtful evaluation setup and clear reporting of negative findings.

Nice efforts are made to make evaluations in geographically and temporally separated way, matching label budgets, context-averaging (both for foundation models and classical learners), and using paired block-based uncertainty estimation. Notably, they carry out an additional BC 2024 replication with frozen protocol, and report a failure to pass the prespecified confirmation rule.

All these things make the empirical analysis more credible. The clear mention of pending comparisons and failed replication is particularly useful as positive findings on foundation models may be quite sensitive to evaluation choices.

  1. Interesting disentangling of the predictive ability and the reliability of response to weather.

The work makes an interesting step beyond the traditional predictive measures, by investigating the variation in weather perturbation response across admissible contexts. The synthetic/sequential/archived FFMC controls serve as a useful benchmark to evaluate the fidelity of response direction.

It is also particularly interesting to see that one can get better predictive fit together with more degraded direction category error. The matched coverage comparison also demonstrates that unanimous context agreement may not be a better way to judge the reliability of directions than the response magnitude.

Critical weaknesses
  1. The main positive base model comparisons do not prove supremacy over the best classical models in the same comparison classes.

As much of the positive BC 2023 comparisons are presented in Section 5.1 and Table 2 and center on the positive results as compared to a logistic regression ensemble (single context validation choice), yet the paper shows 5 context CatBoost and LightGBM models are much better competitors across various label budgets.

TabPFN, TabFM, CatBoost, and LightGBM have nearly the same 5 context recall at 64 label budget, e.g. 11.5-11.6% around. Foundation models have higher point estimates at higher label budgets, but the paper itself explicitly says no direct inferential tests compared to bagged trees for the BC 2023 comparisons.

So much of the significance of the positive advantages is highly dependent on the comparator selection process, so it's difficult to know if the main positive is that positive or if FMs have a real practical advantage when following the recommended evaluation protocol by the authors.

It would go a long way if there are direct paired comparisons shown against 5 context CatBoost and LightGBM (with simultaneous CIs for AP and fixed budget recall etc.).

  1. Little independent evaluation evidence and unclear chronology of protocol freezing.

Only one jurisdiction targeted in external evaluation and 31 forecast initializations dates per season. While the authors rightly use moving block uncertainty estimation, there is considerable overlap in forecast results over time and few independent initialization dates, so there are statistical power / external generalizability concerns.

It's nice to have the BC 2024 replication, but only 2 of 16 simultaneous intervals show positive difference for FM. The rest is unclear, so it's not clear how substantial and/or reliable these seeming advantages are.

One issue is the ambiguity about when things were evaluated: it seems that the BC protocol was frozen prior to the calculation of aggregate BC metrics (Introduction), whereas the local adaptation experiment appears to have been set up after looking at the zero-label result (Section 3). But if the zero label result involves BC 2023 then it's not clear which things were decided upon before looking at BC 2023 outcomes vs. which things were decided after.

That's important because it affects how you'd view the BC 2023 adaptation outcomes as confirming. (Of course this is somewhat buffered because BC 2024 evaluation is separately frozen, but still not clear what the fine-grained chronology looks like.)

Questions for authors

Q1. Clarification and more analysis regarding direct comparison with ensemble baselines.

Please give direct paired concurrent confidence intervals for comparison of 5 context TabPFN vs TabFM and 5 context CatBoost and LightGBM on BC 2023 and BC 2024 (for integrated AP and 5%-alert recall). Also, what if you assess all the classical model candidates with the same 5 context method before choosing your comparison model?

This is the core of your claims about the benefits of foundation models with local supervision.

Q2. Clarification of the timing of evaluation.

Please give a fine-grained chronology of decisions about the protocol and inspection of outcomes. For example, what is the relationship between "we froze the BC protocol before we did any computation of aggregate BC metrics" and "our local adaptation experiment was set up after inspection of the zero label result"?

What things about BC 2023 adaptation experiment were decided before looking at BC 2023 outcomes?

Q3. Clarification and robustness analysis for statistical uncertainty.

How much independent temporal information is there effectively, with 31 dates of initialization per season and so much overlap in forecasting horizons for the 35 day moving block intervals?

Are the CIs you report also capturing the uncertainty due to different independently-sampled contexts sets, or are they conditioning on the 5 that you sample? Does the main results of BC 2024 (based on block-lengths) remain consistent on the shorter block-length sensitivity analysis?

Q4. How weather-response instability should be interpreted. Further clarification/sensitivity analysis.

Out of the 27-28% of contexts with disagreements at 4096 labels, how much of that corresponds to actually different (non-flat) directions of responses? vs. flat/non-flat transitions?

How much the above disagreement percentages depend on the specific choice of threshold for ±0.05 (log-odds)? To understand the nature of the instability observed (important differences in behavior vs. sensitivity to how you categorise the responses).

Reviewer_QeKaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors compare foundation models for tabular data on the task of wildfire forecasting agains CatBoost, LightGBM, logistic regression and the Fine Fuel Moisture Code (FFMC). The data is extracted from events reported in fire journals in 2 regions in Russia and British Columbia. The authors test model transfer capabilities across regions , and using varying amounts of labelled data

Critical strengths

The paper tackles an important challenge of predicting wildfire events and proposes an attemps to evaluate models on various setups including geographical and temporal transfer, and looking into using varying amounts of labeled data for transfer.

Critical weaknesses

My biggest issue is that I found the paper incomprehensible. The entire paper looks LLM generated with no or minimal human edits. I have found this extremely difficult to read and follow. Concepts are mentioned without being defined, we don't understand the point of the different experiments, what the authors are trying to do and then critical information is also missing. We don't understand what the contribution of the authors is. Not once is it mentioned whether the data has been used in other works, what the authors have made differently, if it is actually a new dataset (which I suppose it is?). What are the data sources, the authors mention fire journals, where are they obtained from? Where does the environmental data come from? How many weather variables are considered and what are they? If this information is present, it is so poorly presented and organized that even after reading the data section multiple times I cannot answer these questions. In terms of visualization, if would have been nice to have a visualization of the spatial distribution of the data for example.

The related work section enumerates methods but does not place them in the context of the proposed work of the authors. It makes it really ahrd to assess where that work sits in the landscape of these different models, and what more it brings. Saying “x methods requires y” without saying why “y” might be limiting for example, is not very informative.

the authors do not describe the models they are comparing against. Not once is FFMC described. It is expected if you introduce a word that can refer to many different things like "context" to define what it is.

I could say similar things about each section of the paper but to me good writing is an integral part of the research if you want to communicate your project in a way that other researchers might be able to use this work. Right now, I am very sorry that there might be good ideas but unfortunately they are so buried underneath LLM writing that they don't come through.

I couldn't understand any of the paper.

Questions for authors

I have so many questions to ask related to the critical weaknesses I pointed, I would probably need clarifications at every paragraph to understand the language used in the paper.

But just to at least get a sense of what the paper is trying to do:

  • what are the contributions that go beyond SOTA or what is existing?
  • what is a "context" in this paper? what does it mean to average over contexts for this task? how are they defined?
  • what is case enrichment?
  • what do you mean by "label transfer contrasts?"
  • when you write "An interval containing zero is unresolved; it does not establish equivalence". I don’t even understand what zero refers to here in the task?
  • can you provide more details on the data and fire journals provenance and original format of the data and what transformations to did to it?
  • "None of these tests validates a causal response of real fires to weather." It is not even clear what the original hypothesis of this work is. Can the authors explain why they chose this setup of predicting fires from this data ?
Reviewer_2uRxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares foundation models, including TabPFN and TabFM, with traditional methods for wildfire prediction. It focuses on transfer across regions, the value of a small number of local labels, and changes in weather responses when the context changes. The authors use data from two Russian regions and British Columbia (BC), comparing the foundation models with CatBoost, LightGBM, standard logistic regression, and the FFMC index. They also evaluate on the independent BC 2024 test set. The results do not support an advantage for the foundation models when transferring without local labels. Averaging predictions across contexts helps both foundation models and tree models. In some FFMC reference experiments, lower probability prediction error can coincide with higher error in response direction.

Critical strengths
  1. The authors apply the same context averaging to tree models and foundation models, and use BC 2024 to check the earlier results. The paper clearly reports that the foundation-model advantage did not meet the prespecified criteria for replication across years.
  2. The comparison between CDC and magnitude-based selection is useful. Both rules retain the same proportion of responses, making it possible to assess whether agreement across five contexts is more useful than simply selecting responses with large magnitudes.
Critical weaknesses

After reading the paper, I am still unsure what its main new finding is. Section 2 already discusses work on distribution shifts, context selection, and disagreement in explanations. Sections 5.2 and 5.3 also provide some analysis, including separating the effects of context changes from randomness in model fitting, and varying model capacity and weather inputs. However, these results feel somewhat disconnected, and it remains unclear which findings add to what is already known. The comparison between CDC and the magnitude baseline is more specific, but the paper does not discuss in enough detail what this result means for understanding or using tabular foundation models.

Questions for authors

What do the authors consider the paper's most important finding beyond prior work? Which experiments provide the strongest support for it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.