The Surprising Effectiveness of One-Step Boosting for Tabular Foundation Models

Tabular foundation models (TFMs) predict from labeled examples without updating their parameters, yet those examples can also support a small task-specific correction. We adjust class scores with a validation-selected class-prior shift, then fit ridge regression to cross-fitted probability residuals and add its output to the scores when it improves validation macro-F1. Because the residuals are negative gradients of multiclass cross-entropy, this is one step of supervised boosting. On datasets with uncorrected validation macro-F1 below $0.60$ ($76/77/74$ for TabPFN/TabICL/TabFM), the combined adjustments raise mean test macro-F1 by $6.93/6.50/8.21$ points. The ridge step adds $2.41/3.29/5.16$ points beyond the shift and improves one-vs-rest score-ranking AUC by $2.10/3.58/4.30$ points. Even at validation macro-F1 of $0.60$–$0.80$, it adds $1.44$ macro-F1 and $0.89$ ranking-AUC points on average. On a low-performing TabPFN subset, the ridge gain is $+2.95$ with the full labeled context and $+5.19$ with one quarter of it. On one A100, four-fold cross-fitting and ridge fitting take a median $11.65$s per TabPFN dataset beyond a shared $3.56$s base pass; once backbone scores exist, applying both corrections including test-row featurization takes a separately measured median $31$ms per $1,000$ rows. Thus even capable TFMs benefit from a small, lightweight supervised output update.