Finding Structure Without Fitting Noise: Taking Apart the Small-Table Advantage of Tabular Foundation Models

Tabular foundation models (TFMs) beat tuned gradient boosting and BART on small tables, but an accuracy gap cannot tell finding more structure from fitting less noise. We take a TFM's advantage over a comparator apart with three black-box instruments that need only fit-and-predict calls. Conditional twins keep a table's least-squares fit and residual size and redraw the residual at random, which removes the structure in the residual but keeps everything the linear fit reveals. A structure ablation removes structure estimated on rows outside the context and measures how much of it each learner had found. Noise redraws resample only the noise around that structure and measure what fitting it costs. On 57 OpenML regression tables with 120 context rows, all 11 TFMs we audit have a better mean rank than each of 23 comparators. The four we study in depth find more structure than the tuned boosters on most tables, and more than BART, with the largest gaps on the tables with the most structure: removing the structure costs them more accuracy than BART on 37 of 52 tables, and summed over tables they recover 42% of what it adds to a linear fit against BART's 23%, including structure beyond an additive fit that BART barely uses. They also fit less noise than CV-tuned boosting and LightGBM: the booster's extra cost of fitting noise is 15–20% of their advantage over CV-tuned boosting, and averaging the booster's fold models removes most of it while the TFMs keep 86% of their advantage.