- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether Tabular Foundation Models (TFMs) outperform traditional tree-based methods and Bayesian Additive Regression Trees (BART) because they discover more nonlinear structure or because they fit less noise. The authors propose three black-box methods to investigate these explanations:
- Conditional twins: Generate alternative training datasets by preserving the original linear regression fit and residual magnitude while randomizing the residual direction. The test set remains unchanged. Comparing predictions from the original and randomized datasets measures net discovery, which combines the benefits of structure discovery with the costs of noise fitting.
- Structure ablation: Estimate nonlinear structure using an external regression model and remove it from the original training labels while preserving the linear fit and residual noise. The test set remains unchanged. The increase in test error is used to estimate how much of the removed structure each model had exploited.
- Noise redraws: Preserve the estimated underlying structure while randomizing the residual noise in the training labels. The variance of predictions across different noise realizations is used to measure each model's sensitivity to noise. The authors evaluate 11 TFMs against 23 traditional methods on 57 OpenML regression datasets. Their experiments suggest that TFMs recover more nonlinear structure than traditional methods, particularly on datasets with strong nonlinear relationships. They also exhibit lower noise sensitivity than tuned gradient boosting and LightGBM, although BART is less sensitive to noise than TFMs. Additionally, the advantage of TFMs over boosting decreases as the number of training samples increases.
Critical strengths
- Interesting research question. The paper investigates an important question beyond conventional performance benchmarking. Understanding whether a model's advantage comes from better structure discovery or reduced noise fitting could provide useful insights into when and why different learning algorithms perform well.
- Model-agnostic experimental framework. The proposed methods require only fit-and-predict operations without accessing model internals. This makes the framework relatively straightforward to apply to different tabular models, although structure ablation and noise redraws require additional labeled data.
- Extensive experimental evaluation. The authors evaluate a broad range of TFMs and traditional baselines. They also conduct several robustness checks, including experiments with alternative structure estimators, synthetic datasets with known underlying functions, and different context sizes.
Critical weaknesses
-
Structure ablation does not fully isolate structure discovery. Although the authors preserve the linear regression fit when removing estimated nonlinear structure, this does not guarantee that nonlinear models will learn or utilize the linear component in the same way before and after ablation. Changes in the training labels may also affect regularization, model adaptation, and noise fitting. Therefore, a larger increase in test error does not necessarily imply that a model previously discovered more nonlinear structure. While the authors acknowledge this issue and provide supporting experiments, the interpretation of the ablation gap as a quantitative measure of structure discovery remains somewhat uncertain.
-
The conclusions about structure discovery may be broader than what the experiments directly establish. The experiments primarily demonstrate that TFMs are more sensitive to the removal of estimated nonlinear structure, rather than directly measuring the nonlinear relationships learned by each model. Furthermore, the advantage over BART is concentrated in a relatively small number of datasets. It remains unclear whether TFMs are generally better at discovering nonlinear structure or particularly effective at certain types of relationships.
-
Dependence on the external regression estimator. Both structure ablation and noise redraws rely on an external regression estimator to separate nonlinear structure from noise. If this estimator misses meaningful nonlinear relationships, the estimated residuals may contain both noise and unmodeled structure. Although the authors evaluate alternative estimators and conduct synthetic validation experiments, the sensitivity of the conclusions to estimator misspecification on real-world datasets remains unclear.
-
Limited analysis of how the full decomposition changes with context size. The authors demonstrate that the advantage of TFMs over boosting decreases as the number of training samples increases. They also investigate the conditional-twin decomposition at different context sizes. However, the complete decomposition into linear-fit effects, structure discovery, and noise fitting is primarily evaluated with 120 training samples. Understanding how these contributions change with context size would provide further insights into why traditional methods become more competitive as more training data becomes available.
Questions for authors
- How sensitive are the structure-ablation and noise-redraw results to the accuracy of the external regression estimator? Although the authors evaluate alternative estimators and validate their approach using synthetic datasets, could they provide a more systematic analysis of how estimator quality or misspecification affects the measured structure-discovery and noise-fitting contributions?
- To what extent can the structure-ablation gap be attributed to changes in noise fitting or model adaptation rather than structure discovery? In particular, Appendix I shows that averaging predictions across noise redraws reduces the structure-related gap against CV-tuned boosting. Could the authors further investigate how much of the original ablation gap reflects changes in the learning procedure?
- How does the full decomposition of the performance advantage change as the context size increases? The authors show that TFMs' advantage over boosting decreases with larger context sizes. Is this mainly because traditional methods recover more nonlinear structure, become less sensitive to noise, or improve their handling of linear relationships?
3 comments
Reply to Reviewer gy5u, part 2 of 3: further checks and the estimator (W1, Q2, Q1, W3)
(d) Double the structure. With context y + h, the TFMs' loss changes more than BART's on 28/36 tables (p = 0.001; 27/36 with BART frozen). Against CV-tuned BART, boosting and LightGBM the difference is not significant when they re-tune (19–22/36). With frozen pipelines CV-tuned BART gives 25/36, boosting 24/36 and LightGBM 20/36. So this check separates the TFMs only from BART (Tables 27 and 31).
(e) What the forecasts contain, with no label changed. We regress each learner's real-context forecast, minus the linear fit's, on the estimated structure at the test rows, with Z_t partialled out (Appendix O.4, Table 32). The coefficient mixes amplitude and shape, so we also report the correlation, which depends only on shape. At 120 rows:
| Baseline | Coefficient gap | Correlation gap |
|---|---|---|
| BART | 0.11 [0.06, 0.17], 30/36 | 0.08 [0.04, 0.12], 29/36 |
| CV-tuned boosting | 0.07 [0.04, 0.12], 25/36 | 0.12 [0.08, 0.15], 32/36 |
| Optuna LightGBM | 0.05 [0.02, 0.09], 25/36 | 0.09 [0.06, 0.13], 31/36 |
| CatBoost, early-stopped | 0.09 [0.05, 0.14], 30/36 | 0.10 [0.06, 0.13], 32/36 |
With the MLP estimate the correlation gaps are 0.09–0.16 (28–32/36). Without partialling, the coefficient gap over LightGBM is 0.04 [0.00, 0.08] (20/36; Table 33). Fitted noise lowers a forecast's correlation, which favors the TFMs over the noisier boosters. The conservative measures are therefore the coefficient against CV-tuned boosting and LightGBM (0.05–0.07, 25/36) and the correlation against BART (0.08, 29/36), and both favor the TFMs. This check uses the same f̂, so it is not independent of the ablation.
Your Q2 on Appendix I. Against boosting, the structure part averaged over redraws (0.017, Appendix I) and the noise-free gap (0.040 re-tuned, 0.048 frozen) bracket the paper's 0.029. So we agree that the size of the gap over boosting is not identified. Appendix I of the submission already said that part of it may reflect a change in noise fitting. On the 36 spare-row tables its sign does not change with the check: the mean gap is positive under the rise split, freezing, noise removal and alignment.
Q1 / W3. Sensitivity to the estimator.
Estimator family. Same partitions, with the structure estimated by a GP (ARD), an MLP, a GAM, k-NN, TabPFN 3 and the average of four, each screened on its own audit rows (Appendix O.2). The mean gap over BART is positive with every estimator, and every interval excludes zero (0.012–0.033). Table by table, it is significant for TabPFN 3, GP and MLP (p = 0.003–0.011) and not for trees (0.13), k-NN (0.14), the average (0.18) or the GAM (0.50). Over LightGBM every interval excludes zero, and over CV-tuned boosting every interval except the GAM's.
Estimator quality. With trees fitted on up to 5,000 (the paper's estimate), 1,000 and 250 spare rows, the estimate's worth falls from 0.221 to 0.104 to 0.067. The gap over BART goes 0.033 → 0.026 → 0.022 and stays positive. Over boosting it goes 0.029 → 0.015 → 0.007 [−0.009, 0.020] and vanishes with the weakest estimate. A weak estimate leaves part of the structure in the "noise", which attenuates the measured gap. An exact split of the noise effect points the same way (Appendix O.11): the TFMs do better with the real residual than with a shuffled one (−0.018 [−0.033, −0.006]) and BART does not (−0.003), as expected if the TFMs also use structure that the estimate missed. With the 250-row estimate the TFMs' summed rise is 104% of the estimate's worth, so we read shares only for strong estimates.
The noise half of your Q1 we did not run. The noise redraws were not repeated around the other estimators or the weaker tree estimates, so how the measured noise cost depends on the estimator is untested on real tables. What we have is the planted-truth check of Appendix I, where the error of the estimate inflates the booster's measured extra noise cost (0.020 measured against 0.015 true). We did vary the noise law around the tree estimate (reply to eLKi): the boosters' extra noise cost is positive on 29–34 of 36 tables under all three laws (when the noise is drawn from TabPFN 3's predictive distribution, its mean against CV-tuned boosting is negative because of one table).
Reply to Reviewer gy5u, part 3 of 3: context size and generality (W4, Q3, W2)
Q3 / W4. How the decomposition changes with context size. The full split at 500 to 10,000 rows, against CV-tuned boosting (Appendix O.6). At 5,000 and 10,000 rows the structure is estimated on up to four times as many rows as the context. The table set shrinks as the context grows (36 to 16 tables), so the rows differ in tables as well as in context size:
| Context | Advantage | ΔB | Ablation gap (MSE) | Extra noise cost | Share, TFMs vs boosting |
|---|---|---|---|---|---|
| 120 (Sec. 4.3; App. O.10, O.11) | 0.062, 30/36 | about ⅓ of it | 0.029 [0.013, 0.048], 28/36 | median 0.011, 30/36 | 36% vs 23% |
| 500 | 0.040, 30/31 | 0.015 | 0.007 [−0.009, 0.021], 21/31 | 0.010, 29/31 | 54% vs 51% |
| 1,000 | 0.027, 23/28 | 0.012 | −0.001 [−0.012, 0.008], 13/28 | 0.008, 26/28 | 76% vs 77% |
| 2,000 | 0.059, 22/24 | 0.003 | 0.045 [0.005, 0.116], 16/24 | 0.007, 23/24 | above 100% (small worth; not interpretable) |
| 5,000 | 0.019, 16/20 | 0.003 | 0.009 [−0.004, 0.022], 14/20 | 0.003, 16/20 | 88% vs 81% |
| 10,000 | 0.022, 11/16 | 0.004 | 0.001 [−0.017, 0.017], 9/16 | 0.000, 10/16 | 79% vs 79% |
To answer your question: as context grows, boosting recovers more of the structure (its share reaches the TFMs' by 500 rows) and the linear-fit gap closes (ΔB 0.015 → 0.003). Its extra noise cost shrinks slowly: it stays positive on nearly every table to 2,000 rows and is at most 0.003 by 5,000–10,000 rows, where the advantage itself is small. In normalized MSE the remaining structure gap is small and not significant table by table (p ≥ 0.07). At 500, 1,000 and 2,000 rows the secondary measures still find a small positive gap on most tables: the noise-free ablation (24/31, 21/28 and 18/24 tables), absolute error (post hoc; 25/31, 20/28 and 21/24) and alignment at 500 and 1,000 rows.
At 2,000 rows one table (methane_emissions_rice_crop, gap 0.79) carries the means. Without it, a post hoc exclusion, the ablation gap is 0.013 [0.003, 0.025]. A rerun at 2,000 rows with the structure estimated on up to four times as many rows as the context gives the same picture (0.042 [0.008, 0.101], 18/28 tables). Against BART the gap at 500, 1,000 and 2,000 rows is 0.011, 0.012 and 0.034 (positive on 18/31, 20/28 and 16/24 tables; the sign test is significant only at 1,000 rows). BART goes from fitting less noise than the TFMs at 120 rows to fitting more from 500 rows on (0.007–0.011 at 500 to 2,000 rows, 0.004 at 5,000 and 10,000).
W2. Is the structure edge general? No. It sits on the tables with the most structure.
- Over BART on the paper's 36 spare-row tables, the gap without the five largest gaps is 0.006, as in the submission (Section 4.1). Recomputed with an interval it is 0.007 [−0.002, 0.015] (Appendix O.5).
- On 61 new OpenML tables the gap over BART is positive on 34/61 over all tables, and on 17/21 of the third with the most structure.
- The gap over CV-tuned boosting shows the same pattern there: 37/61 over all tables, with a positive mean (0.034 [0.005, 0.079]; 0.015 [0.002, 0.027] without the one table on which the booster's loss exceeds the target variance), and 16/21 on the top third (Appendix O.12).
- On 16 tables from outside OpenML (CARTE benchmark), where the structure estimate is worth little (0.075), the gap is small: 12/16 over BART (p = 0.077) and 11/16 over boosting.
So the edge appears where the structure estimate adds much to the linear fit, against the boosters and BART alike, and is near zero elsewhere. The abstract now states the claim in that narrower form.
On your point that the ablation shows sensitivity to removal and not what each model learned: check (e) in Part 2 looks at the forecasts directly, with no label changed. On which kinds of relationship matter, the planted-structure experiments of Section 4.1 (already in the submission) show that the edge over BART is larger for interactions than for smooth structure, and reversed for steps with heteroscedastic noise.
Statistics. We report many sign tests and do not correct them for multiplicity. The primary measure is the MSE ablation gap over BART and over CV-tuned boosting, at 120 rows and on the new tables, as a mean with its interval and a sign test over tables. Everything else is descriptive. The absolute-error analysis is post hoc and labeled as such, and summaries that exclude a single heavy-tailed table are secondary.
Reply to Reviewer gy5u, part 1 of 3: does the ablation measure discovery? (W1)
Thank you for the careful review. Your central question is whether a larger rise under the ablation means that more structure was found. New checks say the sign of the gap reflects discovery:
The size of the gap over boosting is not pinned down (0.017 to 0.048). Replies to W3, W4 and W2 follow.
The new checks are in Appendix O of the revised PDF (O.1, O.3 and O.4). The primary measure is the paper's ablation gap in normalized MSE, with a percentile bootstrap interval over tables and a sign test. The checks below are secondary analyses. Notation: P_Z y is the context's linear fit, h is the estimated structure beyond it, and y − h is the ablated context.
(a) Where the rise comes from. With δ the change in a learner's forecast under the ablation, we divide δ into its projection on the removed structure at the test rows (with the test rows' linear part Z_t partialled out) and the rest. The two pieces are orthogonal, so the rise splits exactly into a part along the removed structure and a part orthogonal to it (a changed fit of the linear component, adaptation, changed regularization and everything else). Each learner's own rise is almost all along the structure: 0.077 against 0.003 for the TFMs, 0.043 against 0.004 for BART and 0.056 against −0.005 for CV-tuned boosting. The gaps, on 275 partitions of 36 tables (Appendix O.4, Table 32):
The split does not separate adaptation along the structure, for example a TFM that shrinks harder on a context that looks less structured. So we compared each learner's response to removing the structure with its response to adding it (context y + h), each measured as the coefficient of its forecast change on the structure. The frozen baselines are symmetric (removal minus addition is between −0.001 and 0.012). The TFMs follow added structure more than they lose removed structure (0.369 vs 0.341; difference −0.028 [−0.056, −0.004], negative on 27/36 tables). So their adaptation makes the ablation understate the structure they use, not overstate it.
(b) Freeze the baselines' pipelines. We froze each baseline at its real-context state (the hyperparameters chosen by CV, the target centering and scaling, and for BART the noise-prior scale) and refitted it unchanged on y − h. Freezing changes the rise by −0.001 to 0.005 (Appendix O.3, Table 30). The ablation gaps become: boosting 0.024 [0.009, 0.042] (26/36), LightGBM 0.026 [0.011, 0.042] (28/36), CV-tuned BART 0.022 [0.006, 0.041] (26/36), BART 0.035 [0.011, 0.063] (25/36).
(c) Remove the noise, then ablate. We fitted every learner on the noise-free targets P_Z y + h and P_Z y. Re-tuned, the gap is 0.031 over BART (24/36) and 0.040 over CV-tuned boosting (33/36). Noise-free contexts are out of distribution for every learner, and BART's default noise prior is computed from them. So we also ran the check with every baseline's pipeline frozen at the real context. Frozen, the gap is larger and more consistent against BART and the boosters, and about the same against CV-tuned BART (re-tuned: 0.033, 29/36; Table 31):
Settings chosen on noisy targets over-regularize a baseline on a noiseless one, so only the sign of this check is informative, not its size.