acceptodds
Under review as a conference paper at ICLR 2027

Finding Structure Without Fitting Noise: Taking Apart the Small-Table Advantage of Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) beat tuned gradient boosting and BART on small tables, but an accuracy gap cannot tell finding more structure from fitting less noise. We take a TFM's advantage over a comparator apart with three black-box instruments that need only fit-and-predict calls. Conditional twins keep a table's least-squares fit and residual size and redraw the residual at random, which removes the structure in the residual but keeps everything the linear fit reveals. A structure ablation removes structure estimated on rows outside the context and measures how much of it each learner had found. Noise redraws resample only the noise around that structure and measure what fitting it costs. On 57 OpenML regression tables with 120 context rows, all 11 TFMs we audit have a better mean rank than each of 23 comparators. The four we study in depth find more structure than the tuned boosters on most tables, and more than BART, with the largest gaps on the tables with the most structure: removing the structure costs them more accuracy than BART on 37 of 52 tables, and summed over tables they recover 42% of what it adds to a linear fit against BART's 23%, including structure beyond an additive fit that BART barely uses. They also fit less noise than CV-tuned boosting and LightGBM: the booster's extra cost of fitting noise is 15–20% of their advantage over CV-tuned boosting, and averaging the booster's fold models removes most of it while the TFMs keep 86% of their advantage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, tabular data, in-context learning, bias-variance decomposition, co-sufficient sampling, rotation tests, structure ablation, gradient boosting, BART

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gy5uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether Tabular Foundation Models (TFMs) outperform traditional tree-based methods and Bayesian Additive Regression Trees (BART) because they discover more nonlinear structure or because they fit less noise. The authors propose three black-box methods to investigate these explanations:

  1. Conditional twins: Generate alternative training datasets by preserving the original linear regression fit and residual magnitude while randomizing the residual direction. The test set remains unchanged. Comparing predictions from the original and randomized datasets measures net discovery, which combines the benefits of structure discovery with the costs of noise fitting.
  2. Structure ablation: Estimate nonlinear structure using an external regression model and remove it from the original training labels while preserving the linear fit and residual noise. The test set remains unchanged. The increase in test error is used to estimate how much of the removed structure each model had exploited.
  3. Noise redraws: Preserve the estimated underlying structure while randomizing the residual noise in the training labels. The variance of predictions across different noise realizations is used to measure each model's sensitivity to noise. The authors evaluate 11 TFMs against 23 traditional methods on 57 OpenML regression datasets. Their experiments suggest that TFMs recover more nonlinear structure than traditional methods, particularly on datasets with strong nonlinear relationships. They also exhibit lower noise sensitivity than tuned gradient boosting and LightGBM, although BART is less sensitive to noise than TFMs. Additionally, the advantage of TFMs over boosting decreases as the number of training samples increases.
Critical strengths
  1. Interesting research question. The paper investigates an important question beyond conventional performance benchmarking. Understanding whether a model's advantage comes from better structure discovery or reduced noise fitting could provide useful insights into when and why different learning algorithms perform well.
  2. Model-agnostic experimental framework. The proposed methods require only fit-and-predict operations without accessing model internals. This makes the framework relatively straightforward to apply to different tabular models, although structure ablation and noise redraws require additional labeled data.
  3. Extensive experimental evaluation. The authors evaluate a broad range of TFMs and traditional baselines. They also conduct several robustness checks, including experiments with alternative structure estimators, synthetic datasets with known underlying functions, and different context sizes.
Critical weaknesses
  1. Structure ablation does not fully isolate structure discovery. Although the authors preserve the linear regression fit when removing estimated nonlinear structure, this does not guarantee that nonlinear models will learn or utilize the linear component in the same way before and after ablation. Changes in the training labels may also affect regularization, model adaptation, and noise fitting. Therefore, a larger increase in test error does not necessarily imply that a model previously discovered more nonlinear structure. While the authors acknowledge this issue and provide supporting experiments, the interpretation of the ablation gap as a quantitative measure of structure discovery remains somewhat uncertain.

  2. The conclusions about structure discovery may be broader than what the experiments directly establish. The experiments primarily demonstrate that TFMs are more sensitive to the removal of estimated nonlinear structure, rather than directly measuring the nonlinear relationships learned by each model. Furthermore, the advantage over BART is concentrated in a relatively small number of datasets. It remains unclear whether TFMs are generally better at discovering nonlinear structure or particularly effective at certain types of relationships.

  3. Dependence on the external regression estimator. Both structure ablation and noise redraws rely on an external regression estimator to separate nonlinear structure from noise. If this estimator misses meaningful nonlinear relationships, the estimated residuals may contain both noise and unmodeled structure. Although the authors evaluate alternative estimators and conduct synthetic validation experiments, the sensitivity of the conclusions to estimator misspecification on real-world datasets remains unclear.

  4. Limited analysis of how the full decomposition changes with context size. The authors demonstrate that the advantage of TFMs over boosting decreases as the number of training samples increases. They also investigate the conditional-twin decomposition at different context sizes. However, the complete decomposition into linear-fit effects, structure discovery, and noise fitting is primarily evaluated with 120 training samples. Understanding how these contributions change with context size would provide further insights into why traditional methods become more competitive as more training data becomes available.

Questions for authors
  1. How sensitive are the structure-ablation and noise-redraw results to the accuracy of the external regression estimator? Although the authors evaluate alternative estimators and validate their approach using synthetic datasets, could they provide a more systematic analysis of how estimator quality or misspecification affects the measured structure-discovery and noise-fitting contributions?
  2. To what extent can the structure-ablation gap be attributed to changes in noise fitting or model adaptation rather than structure discovery? In particular, Appendix I shows that averaging predictions across noise redraws reduces the structure-related gap against CV-tuned boosting. Could the authors further investigate how much of the original ablation gap reflects changes in the learning procedure?
  3. How does the full decomposition of the performance advantage change as the context size increases? The authors show that TFMs' advantage over boosting decreases with larger context sizes. Is this mainly because traditional methods recover more nonlinear structure, become less sensitive to noise, or improve their handling of linear relationships?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer gy5u, part 1 of 3: does the ablation measure discovery? (W1)

Thank you for the careful review. Your central question is whether a larger rise under the ablation means that more structure was found. New checks say the sign of the gap reflects discovery:

  • The TFMs' extra rise lies along the removed structure (0.019–0.034 over the baselines, 27–30/36 tables). With the paper's tree estimate the rest is small (−0.001 to 0.008, and every interval includes zero).
  • Freezing the baselines' pipelines changes their rise by at most 0.005.
  • With the noise removed and the pipelines frozen, the gap is positive on 30–35/36 tables.
  • With no label changed, the TFMs' forecasts correlate more with the structure (29–32/36 tables).

The size of the gap over boosting is not pinned down (0.017 to 0.048). Replies to W3, W4 and W2 follow.

The new checks are in Appendix O of the revised PDF (O.1, O.3 and O.4). The primary measure is the paper's ablation gap in normalized MSE, with a percentile bootstrap interval over tables and a sign test. The checks below are secondary analyses. Notation: P_Z y is the context's linear fit, h is the estimated structure beyond it, and y − h is the ablated context.

(a) Where the rise comes from. With δ the change in a learner's forecast under the ablation, we divide δ into its projection on the removed structure at the test rows (with the test rows' linear part Z_t partialled out) and the rest. The two pieces are orthogonal, so the rise splits exactly into a part along the removed structure and a part orthogonal to it (a changed fit of the linear component, adaptation, changed regularization and everything else). Each learner's own rise is almost all along the structure: 0.077 against 0.003 for the TFMs, 0.043 against 0.004 for BART and 0.056 against −0.005 for CV-tuned boosting. The gaps, on 275 partitions of 36 tables (Appendix O.4, Table 32):

BaselineRise gap along the structureRise gap, rest
BART0.034 [0.015, 0.056], 30/36−0.001 [−0.008, 0.007], 17/36
CV-tuned BART0.020 [0.009, 0.036], 30/360.004 [−0.004, 0.012]
CV-tuned boosting0.021 [0.012, 0.032], 29/360.008 [−0.002, 0.018]
Optuna LightGBM0.019 [0.010, 0.030], 27/360.006 [−0.005, 0.016]
BART, boosting, LightGBM frozen (see b)0.021–0.034, 31–32/360.001–0.005, intervals include zero
MLP estimate instead of trees0.014–0.025, 27–32/360.006–0.017 (LightGBM: 0.017 [0.004, 0.031], as large as its part along the structure)

The split does not separate adaptation along the structure, for example a TFM that shrinks harder on a context that looks less structured. So we compared each learner's response to removing the structure with its response to adding it (context y + h), each measured as the coefficient of its forecast change on the structure. The frozen baselines are symmetric (removal minus addition is between −0.001 and 0.012). The TFMs follow added structure more than they lose removed structure (0.369 vs 0.341; difference −0.028 [−0.056, −0.004], negative on 27/36 tables). So their adaptation makes the ablation understate the structure they use, not overstate it.

(b) Freeze the baselines' pipelines. We froze each baseline at its real-context state (the hyperparameters chosen by CV, the target centering and scaling, and for BART the noise-prior scale) and refitted it unchanged on y − h. Freezing changes the rise by −0.001 to 0.005 (Appendix O.3, Table 30). The ablation gaps become: boosting 0.024 [0.009, 0.042] (26/36), LightGBM 0.026 [0.011, 0.042] (28/36), CV-tuned BART 0.022 [0.006, 0.041] (26/36), BART 0.035 [0.011, 0.063] (25/36).

(c) Remove the noise, then ablate. We fitted every learner on the noise-free targets P_Z y + h and P_Z y. Re-tuned, the gap is 0.031 over BART (24/36) and 0.040 over CV-tuned boosting (33/36). Noise-free contexts are out of distribution for every learner, and BART's default noise prior is computed from them. So we also ran the check with every baseline's pipeline frozen at the real context. Frozen, the gap is larger and more consistent against BART and the boosters, and about the same against CV-tuned BART (re-tuned: 0.033, 29/36; Table 31):

  • BART: 0.040 [0.016, 0.067], 30/36 tables
  • CV-tuned BART: 0.032 [0.010, 0.054], 30/36
  • CV-tuned boosting: 0.048 [0.029, 0.072], 35/36
  • LightGBM: 0.053 [0.035, 0.074], 35/36

Settings chosen on noisy targets over-regularize a baseline on a noiseless one, so only the sign of this check is informative, not its size.

AuthorsOpenReview ↗

Reply to Reviewer gy5u, part 2 of 3: further checks and the estimator (W1, Q2, Q1, W3)

(d) Double the structure. With context y + h, the TFMs' loss changes more than BART's on 28/36 tables (p = 0.001; 27/36 with BART frozen). Against CV-tuned BART, boosting and LightGBM the difference is not significant when they re-tune (19–22/36). With frozen pipelines CV-tuned BART gives 25/36, boosting 24/36 and LightGBM 20/36. So this check separates the TFMs only from BART (Tables 27 and 31).

(e) What the forecasts contain, with no label changed. We regress each learner's real-context forecast, minus the linear fit's, on the estimated structure at the test rows, with Z_t partialled out (Appendix O.4, Table 32). The coefficient mixes amplitude and shape, so we also report the correlation, which depends only on shape. At 120 rows:

BaselineCoefficient gapCorrelation gap
BART0.11 [0.06, 0.17], 30/360.08 [0.04, 0.12], 29/36
CV-tuned boosting0.07 [0.04, 0.12], 25/360.12 [0.08, 0.15], 32/36
Optuna LightGBM0.05 [0.02, 0.09], 25/360.09 [0.06, 0.13], 31/36
CatBoost, early-stopped0.09 [0.05, 0.14], 30/360.10 [0.06, 0.13], 32/36

With the MLP estimate the correlation gaps are 0.09–0.16 (28–32/36). Without partialling, the coefficient gap over LightGBM is 0.04 [0.00, 0.08] (20/36; Table 33). Fitted noise lowers a forecast's correlation, which favors the TFMs over the noisier boosters. The conservative measures are therefore the coefficient against CV-tuned boosting and LightGBM (0.05–0.07, 25/36) and the correlation against BART (0.08, 29/36), and both favor the TFMs. This check uses the same f̂, so it is not independent of the ablation.

Your Q2 on Appendix I. Against boosting, the structure part averaged over redraws (0.017, Appendix I) and the noise-free gap (0.040 re-tuned, 0.048 frozen) bracket the paper's 0.029. So we agree that the size of the gap over boosting is not identified. Appendix I of the submission already said that part of it may reflect a change in noise fitting. On the 36 spare-row tables its sign does not change with the check: the mean gap is positive under the rise split, freezing, noise removal and alignment.

Q1 / W3. Sensitivity to the estimator.

Estimator family. Same partitions, with the structure estimated by a GP (ARD), an MLP, a GAM, k-NN, TabPFN 3 and the average of four, each screened on its own audit rows (Appendix O.2). The mean gap over BART is positive with every estimator, and every interval excludes zero (0.012–0.033). Table by table, it is significant for TabPFN 3, GP and MLP (p = 0.003–0.011) and not for trees (0.13), k-NN (0.14), the average (0.18) or the GAM (0.50). Over LightGBM every interval excludes zero, and over CV-tuned boosting every interval except the GAM's.

Estimator quality. With trees fitted on up to 5,000 (the paper's estimate), 1,000 and 250 spare rows, the estimate's worth falls from 0.221 to 0.104 to 0.067. The gap over BART goes 0.033 → 0.026 → 0.022 and stays positive. Over boosting it goes 0.029 → 0.015 → 0.007 [−0.009, 0.020] and vanishes with the weakest estimate. A weak estimate leaves part of the structure in the "noise", which attenuates the measured gap. An exact split of the noise effect points the same way (Appendix O.11): the TFMs do better with the real residual than with a shuffled one (−0.018 [−0.033, −0.006]) and BART does not (−0.003), as expected if the TFMs also use structure that the estimate missed. With the 250-row estimate the TFMs' summed rise is 104% of the estimate's worth, so we read shares only for strong estimates.

The noise half of your Q1 we did not run. The noise redraws were not repeated around the other estimators or the weaker tree estimates, so how the measured noise cost depends on the estimator is untested on real tables. What we have is the planted-truth check of Appendix I, where the error of the estimate inflates the booster's measured extra noise cost (0.020 measured against 0.015 true). We did vary the noise law around the tree estimate (reply to eLKi): the boosters' extra noise cost is positive on 29–34 of 36 tables under all three laws (when the noise is drawn from TabPFN 3's predictive distribution, its mean against CV-tuned boosting is negative because of one table).

AuthorsOpenReview ↗

Reply to Reviewer gy5u, part 3 of 3: context size and generality (W4, Q3, W2)

Q3 / W4. How the decomposition changes with context size. The full split at 500 to 10,000 rows, against CV-tuned boosting (Appendix O.6). At 5,000 and 10,000 rows the structure is estimated on up to four times as many rows as the context. The table set shrinks as the context grows (36 to 16 tables), so the rows differ in tables as well as in context size:

ContextAdvantageΔBAblation gap (MSE)Extra noise costShare, TFMs vs boosting
120 (Sec. 4.3; App. O.10, O.11)0.062, 30/36about ⅓ of it0.029 [0.013, 0.048], 28/36median 0.011, 30/3636% vs 23%
5000.040, 30/310.0150.007 [−0.009, 0.021], 21/310.010, 29/3154% vs 51%
1,0000.027, 23/280.012−0.001 [−0.012, 0.008], 13/280.008, 26/2876% vs 77%
2,0000.059, 22/240.0030.045 [0.005, 0.116], 16/240.007, 23/24above 100% (small worth; not interpretable)
5,0000.019, 16/200.0030.009 [−0.004, 0.022], 14/200.003, 16/2088% vs 81%
10,0000.022, 11/160.0040.001 [−0.017, 0.017], 9/160.000, 10/1679% vs 79%

To answer your question: as context grows, boosting recovers more of the structure (its share reaches the TFMs' by 500 rows) and the linear-fit gap closes (ΔB 0.015 → 0.003). Its extra noise cost shrinks slowly: it stays positive on nearly every table to 2,000 rows and is at most 0.003 by 5,000–10,000 rows, where the advantage itself is small. In normalized MSE the remaining structure gap is small and not significant table by table (p ≥ 0.07). At 500, 1,000 and 2,000 rows the secondary measures still find a small positive gap on most tables: the noise-free ablation (24/31, 21/28 and 18/24 tables), absolute error (post hoc; 25/31, 20/28 and 21/24) and alignment at 500 and 1,000 rows.

At 2,000 rows one table (methane_emissions_rice_crop, gap 0.79) carries the means. Without it, a post hoc exclusion, the ablation gap is 0.013 [0.003, 0.025]. A rerun at 2,000 rows with the structure estimated on up to four times as many rows as the context gives the same picture (0.042 [0.008, 0.101], 18/28 tables). Against BART the gap at 500, 1,000 and 2,000 rows is 0.011, 0.012 and 0.034 (positive on 18/31, 20/28 and 16/24 tables; the sign test is significant only at 1,000 rows). BART goes from fitting less noise than the TFMs at 120 rows to fitting more from 500 rows on (0.007–0.011 at 500 to 2,000 rows, 0.004 at 5,000 and 10,000).

W2. Is the structure edge general? No. It sits on the tables with the most structure.

  • Over BART on the paper's 36 spare-row tables, the gap without the five largest gaps is 0.006, as in the submission (Section 4.1). Recomputed with an interval it is 0.007 [−0.002, 0.015] (Appendix O.5).
  • On 61 new OpenML tables the gap over BART is positive on 34/61 over all tables, and on 17/21 of the third with the most structure.
  • The gap over CV-tuned boosting shows the same pattern there: 37/61 over all tables, with a positive mean (0.034 [0.005, 0.079]; 0.015 [0.002, 0.027] without the one table on which the booster's loss exceeds the target variance), and 16/21 on the top third (Appendix O.12).
  • On 16 tables from outside OpenML (CARTE benchmark), where the structure estimate is worth little (0.075), the gap is small: 12/16 over BART (p = 0.077) and 11/16 over boosting.

So the edge appears where the structure estimate adds much to the linear fit, against the boosters and BART alike, and is near zero elsewhere. The abstract now states the claim in that narrower form.

On your point that the ablation shows sensitivity to removal and not what each model learned: check (e) in Part 2 looks at the forecasts directly, with no label changed. On which kinds of relationship matter, the planted-structure experiments of Section 4.1 (already in the submission) show that the edge over BART is larger for interactions than for smooth structure, and reversed for steps with heteroscedastic noise.

Statistics. We report many sign tests and do not correct them for multiplicity. The primary measure is the MSE ablation gap over BART and over CV-tuned boosting, at 120 rows and on the new tables, as a mean with its interval and a sign test over tables. Everything else is descriptive. The absolute-error analysis is post hoc and labeled as such, and summaries that exclude a single heavy-tailed table are secondary.

Reviewer_eLKiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper analyzes where the gains of tabular foundation models over tree-based models come from on small datasets, by decomposing them into the estimation of linear structure, nonlinear structure, and the resilience to noise. It does so using different types of experiments, and concludes that the gains come from both linear and nonlinear structure, and in some cases, also noise.

Critical strengths
  • The paper's main question of understanding and decomposing the gains of tabular foundation models is interesting, the methodology is also somewhat interesting, and the experiments are comprehensive, although I would say the potential impact is moderate.
Critical weaknesses
  • The writing is clearly LLM-ish throughout the paper, which makes the reading harder and not very pleasant. This includes unconventional naming, such as "comparators" instead of "baselines", "roster" (??), or "context-tuned" instead of "holdout-tuned".
  • I think the evaluation design is a bit complicated / unconventional and could be improved. For example, in Section 2.2, the current DD mixes noise cost and structure estimation. Instead of Lt(pˉ)L_t(\bar p), wouldn't it be better to use K−1∑k≥1Lt(pk)K^{-1} \sum_{k \geq 1} L_t(p_k) to disentangle the two? Going even further, the linear regression in Section 2.1/2.2 is fit on the context set while the structure model in Section 2.3 is fit on the "spare rows". Why not fit both on the spare rows and then decompose y = linear + nonlinear + noise and fit the tabular models on subsets of the three? You could fit the tabular models on all subsets of the three and get SHAP-like scores, or choose a particular order like {linear, linear + nonlinear, linear + nonlinear + noise} that would (as in the SHAP-like case) leave no remainder, or use {y - noise, y - linear, y - nonlinear} as in an ablation. If you fit a TFM on the spare rows you could even re-draw the noise in a proper heteroskedastic way by drawing it from the TFM's predictive distribution (and potentially analyze what effect that has vs the shuffling which makes it homoskedastic), though when fitting a TFM on the spare rows you might have to argue why that isn't advantageous for the context-fitted TFMs in comparison to the tree-based methods. And in either case, when you redraw the noise, as you say you are measuring the variance, but why not also measure the bias? (e.g., a comparison of the averaged noise-fitted model to the no-noise-fitted model.) And by the way, it has been argued that the benefit of some tabular models can be explained by their handling of heteroskedastic noise (https://arxiv.org/abs/2509.04430), so it could be interesting to analyze this here too.
Questions for authors
  • Why use only one estimator for the main TFMs? And since the context is small (120 samples), why not use repeated CV, such as the 5-repeated 5-fold CV used in BeyondArena for tiny datasets? This should yield a more fair comparison of what is achievable with TFMs and tree-based methods.
  • ll. 289: Why is the preprocessing equalized between learners, rather than using the learner's preferred preprocessing? For example, TFMs are specifically pre-trained to handle categoricals through ordinal encoding internally, and for tree-based methods, direct categorical handling is also usually applied instead of one-hot encoding.
  • l. 321: Why is BART context-tuned and not CV-tuned?

Minor comments:

  • Section 2.2: Using pp to denote regression predictions is probably misleading.
  • l. 430: "If its extra noise cost is noise" ??
  • l. 042: "a comparator's excess noise can be averaged away" - how?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer eLKi, part 1 of 2: your design, bias, and the writing

Thank you for the review and for the concrete design. We ran a two-factor version of it, and it is now in the main text (Section 2.4). Fitting every learner on y, y − h, P_Z y + h and P_Z y splits the TFMs' advantage into a linear-only part, a structure part and a noise part with no remainder. Against CV-tuned boosting the structure part is 0.035 (31/36 tables) and the noise part 0.034 (32/36). Against BART the advantage is mostly structure (0.032). Against boosting with 5×5 repeated CV and bagging, BART with repeated CV, and boosters with native categorical handling, the TFMs still win on at least 28/36 tables. We also rewrote the paper in plainer language and replaced the terms you flagged.

Your design. The context targets are split into a linear part P_Z y, the structure beyond it h = (I − P_Z) f̂, with f̂ fitted on the spare rows, and the rest (noise). The Shapley values of the two factors give the split. Unlike in your proposal, the linear part stays in all four targets. It is the context's own least-squares fit, which the twins and the ablation hold fixed, and not a fit on the spare rows. On the 36 spare-row tables (275 partitions; Appendix O.1):

BaselineAdvantageLinear onlyStructureNoise
BART0.055 [0.029, 0.084]0.0130.032 [0.011, 0.058], 24/360.010, 25/36
CV-tuned BART0.0400.0070.029 [0.014, 0.047], 30/360.004
CV-tuned boosting0.062 [0.037, 0.090]−0.0060.035 [0.019, 0.054], 31/360.034 [0.019, 0.050], 32/36
Optuna LightGBM0.059−0.0030.031, 31/360.031, 28/36

How this relates to Eq. 6:

  • The structure parts lie between the ablation gaps with and without noise in the context.
  • The noise part of this split is the whole effect of the context's noise on test loss. Eq. 6's extra noise cost is only the forecast variance across noise redraws. Against boosting, the noise part is 0.034 of a 0.062 advantage, and the linear-only part is −0.006: on P_Z y, boosting matches the TFMs. Hence much of ΔB against the boosters reflects how the residual noise distorts the trees' fit of a linear trend.
  • Against BART the two measures seem to disagree: the noise part is positive (0.010, 25/36 tables), yet BART's forecasts vary less across redraws and are no more biased by them. An exact split of the noise effect resolves this (Appendix O.11). The variance and squared-bias terms favor BART (−0.006 and −0.008). What favors the TFMs is a term the redraws average out, how a learner does with the real residual against a shuffled one: 0.015 [0.004, 0.027], 26/36 tables. Shuffling destroys any structure the tree estimate missed and any heteroskedasticity, so this term is what one would see if the TFMs also used structure left in the residual.

D mixes noise cost and structure. We agree, and the paper relies on the ablation and the noise redraws to separate the two. Your averaged twin loss, K⁻¹ Σ_k L(ŷ_k), is our structure gain G in Appendix A, and G = D + V exactly, where V is the forecast variance across twins. G mixes the two as well: for a linear smoother it overstates discovery by a share of V, and D understates it (Proposition 2). We keep the twins because they need no spare rows and so run on all 57 tables.

Bias as well as variance. As you proposed, we measured the squared distance between a learner's forecast averaged over the redraws and its forecast from the noise-free target P_Z y + h, minus the Monte Carlo variance of a nine-redraw mean. The extra squared bias (baseline minus TFMs; Appendix O.11) is:

  • LightGBM: 0.009 [0.004, 0.014], 29/36 tables with permutation draws, and 0.009 [0.005, 0.013], 31/36 with the TabPFN 3 noise law (Part 2)
  • CV-tuned boosting: 0.002–0.003, intervals include zero
  • BART: −0.008 [−0.016, 0.003] (BART is no more biased)
  • CV-tuned BART: 0.012 [0.000, 0.031]

Writing and naming. The introduction is in plainer language, without the slogans and the rhetorical questions. In Sections 2–6 and the appendices we split the long sentences and replaced shorthand with the plain statement. Section 2.3 defines each quantity in a plain sentence (the rise, the ablation gap, the structure's worth), and Section 3 says that "CV-tuned boosting" is the cross-validated scikit-learn histogram gradient boosting. The language pass changed no number and no claim; the changes of substance are listed in the general response. "Comparators" is now "baselines", "roster" is now "table set", and forecasts are written ŷ instead of p throughout. "Context-tuned" was our worst term. For BART it always meant tuned by 5-fold cross-validation on the context rows, so it is now "CV-tuned BART". This answers your l. 321 question. For CatBoost it meant early stopping on a validation split of the context rows, so it is now "early-stopped CatBoost".

AuthorsOpenReview ↗

Reply to Reviewer eLKi, part 2 of 2: noise, baselines and the remaining questions

Heteroskedastic noise from a predictive distribution. For each context row, the noise is drawn from TabPFN 3's predictive distribution, fitted on the spare rows, so each row keeps its own predicted noise scale and shape. We also re-ran it with a noise law that does not come from a TFM: boosted quantile regressions at 39 levels. With that law the boosters' extra noise cost is larger and more consistent: 0.018 [0.014, 0.023] for CV-tuned boosting (34/36 tables) and 0.015 [0.005, 0.022] for LightGBM (33/36). BART again fits less noise than the TFMs (its extra noise cost is positive on 12/36 tables).

BaselineExtra noise cost, TabPFN 3 law (median, tables)Same, permutation
CV-tuned boosting0.009, 29/360.011, 30/36
Optuna LightGBM0.012, 30/360.013, 33/36
BART−0.006, 13/36 (BART fits less)−0.008, 14/36

Table by table, the picture is unchanged, but one table changes the means. On methane_emissions_rice_crop the TFMs' noise cost under the TabPFN 3 law is 0.91, against 0.16 for boosting. This flips the mean extra noise cost against boosting to −0.009 [−0.054, 0.017]. So with strongly heteroskedastic, heavy-tailed noise, the TFMs can be the more noise-sensitive learners, which fits the data-uncertainty account of Kartashev et al. (arXiv 2509.04430, now cited).

Repeated CV. We re-tuned CV-tuned boosting with 5×5 repeated CV over the same grid, refitted on all context rows or bagged over the 25 fold models (Appendix O.10):

BaselineTFMs' advantageTables wonAblation gap
Boosting, 5-fold CV (paper)0.062 [0.037, 0.090]30/360.029 [0.013, 0.048], 28/36
Boosting, 5×5 repeated CV0.054 [0.024, 0.084]29/360.025 [0.009, 0.043], 25/36
Same, bagged over 25 folds0.047 [0.016, 0.075]28/360.025 [0.008, 0.045], 27/36
BART, CV-tuned, 5×5 repeated CV0.035 [0.014, 0.055]28/360.023 [0.007, 0.041], 26/36
BART, CV-tuned over a wider grid0.052 [0.024, 0.077]30/360.017 [0.003, 0.030], 24/36

Repeated CV and bagging cut the TFMs' advantage over boosting by up to a quarter, but the TFMs still win on at least 28 of 36 tables. Repeated CV lowers CV-tuned BART's loss by 0.004 (advantage 0.040 → 0.035), and the ablation gap stays positive. The wider BART grid (24 configurations; submitted Appendix G) does no better than the paper's CV-tuned BART: choosing among 24 configurations by 5-fold CV on 120 rows is itself a fit to the context.

One estimator for the main TFMs. We audit single estimators so that each forecast is a deterministic function of the context. Then the twins and the noise redraws measure the learner, not the randomness of an ensemble's feature and target transforms. Appendix I runs the four main TFMs with their default ensembles and native pipelines on the same twins. With TabPFN 3 and TabICLv2 native, the ablation gap over BART is 0.032 [0.008, 0.062] (Appendix B), against 0.033 for single estimators. With all four native, the net-discovery gap over raw-feature BART is 0.024 [−0.001, 0.052].

Equalized preprocessing. We equalize preprocessing so that differences in loss can be attributed to the learners, and the twins need one fixed feature matrix with a linear basis.

New: we refitted the three boosters on the raw columns with their own categorical and missing-value handling (Appendix O.10). Their mean loss falls by 0.026–0.044, most of it on a few tables. The TFMs, still on the equalized features, win on 28/36 tables against each. Their mean advantage falls to 0.025–0.036, with intervals that include zero. The structure part is unchanged (0.028–0.031, 30–32/36 tables), and the noise part is smaller (0.034 → 0.019 against CV-tuned boosting).

Also new: on three full-width tables, without the 20-column cap, the ablation gap is positive on all three against every baseline, and on two of three with the TFMs in their native pipelines (Appendix O.8).

Minor comments.

  • Section 2.2: forecasts are now ŷ_k, and the twin average is ŷ with a bar.
  • l. 430 now reads: "If its extra noise cost comes from fitted noise and not from structure, averaging models that saw different subsamples should remove it and leave the ablation gap."
  • l. 042: we now say how. Averaging models fitted to different subsamples, such as a learner's cross-validation fold models, reduces the noise cost but cannot recover structure the learner missed (Section 4.2).

Impact. Against boosters, about half of the TFMs' small-table advantage is the effect of noise. Averaging removes most of the variance part of it (15–20% of the advantage), so bagging helps the boosters but closes only part of the gap. Against BART the advantage is mostly structure, and BART has no extra noise cost to average away. The instruments say which case applies for any black-box learner.

Thank you for the detailed response. Here are a few comments, I'll try to write a more general response later:

We equalize preprocessing so that differences in loss can be attributed to the learners, and the twins need one fixed feature matrix with a linear basis.

Is it not possible to just fit the twins with a fixed "linear model" preprocessing (that should probably use one-hot encoding) and use a separate preprocessing for the other models (that uses their own preferred categorical encoding)? I'm worried here, also given the results in your response, that your results are only valid for a setting that most people wouldn't actually use.

We audit single estimators so that each forecast is a deterministic function of the context. Then the twins and the noise redraws measure the learner, not the randomness of an ensemble's feature and target transforms

If you fix a random seed, then it's also deterministic with more estimators.

"table set" is still unusual terminology - datasets or benchmark (or "set of tables") would be more standard.

Reviewer_sb9bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes three black-box instruments, conditional twins, structure ablation and noise redraws, to decide whether a learner's accuracy advantage on small tables comes from recovering more of the table's structure or from fitting less of its noise, and applies them to tabular foundation models against BART and tuned gradient boosting. Twins keep a table's linear fit and residual size and redraw the residual at random; the ablation removes structure estimated on held-out rows and measures the accuracy each learner loses; noise redraws resample only the noise around that structure and measure what fitting it costs. The experiments cover 57 OpenML regression tables at 120 context rows with at most 20 features, eleven TFMs and 23 comparators. The authors report that the TFMs recover a larger share of the estimated structure than BART and the tuned boosters, fit less noise than CV-tuned boosting, and keep most of their advantage once the booster's noise is averaged away. The instruments are sound and the ablations are carefully done, but the evidence is confined to regression, 20 features and 120 context rows, and the paper's own appendices show the effect shrinking to zero as the context grows. This is too narrow a base to validate the general claim, and a more exhaustive study on larger and higher-dimensional tables, and ideally classification, is needed.

Critical strengths
  1. The question is clearly posed and the decomposition is clean. The paper states precisely what an accuracy gap cannot tell (L040-L042) and sets up each instrument to answer one question (L043-L053, Fig. 3). The split of the advantage into a linear-fit part ΔB\Delta B, an extra noise cost, an ablation gap and a remainder (Eq. 6, Fig. 1) is easy to follow, and the authors are explicit about which parts are exact identities and which are measurements (L269-L270). This is consequential because the methodological contribution stands regardless of how far the empirical claims generalise.

  2. The instruments are black-box and inexpensive. Each audit needs only K+1K+1 fit-and-predict calls (L199-L200), and Appendix N shows that 9 to 19 twins reproduce the 39-twin results (L2134-L2140), so the method can be applied to other learners without access to their internals.

  3. The ablations and controls are adequately done within the chosen setup. Planted-truth checks (App. A, B, I), a placebo (Table 5, L870), a prior-family negative control (Fig. 5d), an alternative structure estimator f^\hat{f} (Table 2) and multiple partitions (App. K) are present, and unfavourable results are reported rather than omitted (L334, L353-L355, L480-L484). This supports the view that the concerns raised below are about scope rather than about the correctness of what was run.

Critical weaknesses
  1. The evidence base is too narrow for the claims made. The headline findings (L020-L030, L090-L104) are established at 120 context rows, at most 20 features and regression only. Table 15 shows several tables truncated to 20 columns from 32 to 122 raw features, and classification is excluded because exact twins do not exist for 0/1 targets (L485-L486). The authors' own appendices show the effect disappearing as the context grows: the net-discovery gap over CV-tuned boosting falls from 0.086 at 60 rows to 0.000 at 1,000 (Table 22), the ablation gap over tuned LightGBM is not detected at 1,000 rows (L963-L967), and the advantage over LightGBM is 0.001 [-0.008, 0.014] at 10,000 rows (L2048). The thesis therefore holds only in a narrow regime that the paper itself shows to be narrow, while the abstract and introduction are written generally. This is the main consideration behind my recommendation: unless the study is extended to substantially larger and higher-dimensional tables, and ideally to classification, the current evidence does not support the general claim.

  2. All tables come from a single source. The 57 roster tables and the 16-table replication set are drawn exclusively from OpenML (L282-L288, Table 15); no tables from any other repository or benchmark family are used. This matters twice over. First, it limits how far the thesis can be generalised beyond the OpenML regression population. Second, OpenML is also the main pretraining source for several of the audited TFMs: only 34 of the 57 tables were screened against listed pretraining corpora, the remaining 23 "widely used" tables were not (L282-L284), and TabDPT-Turbo's corpus is described only by a rule over OpenML rather than table by table (L285-L286). An independent dataset family would address both concerns at once.

  3. The ablation measures discovery only within what the structure estimator finds, and that estimator is tree-based. f^\hat{f} averages a random forest and gradient boosting (L205), and the authors acknowledge that discovery is measured within what f^\hat{f} captures (L484-L485). Since the main comparators are also tree ensembles, the shared inductive bias confounds the comparison in both directions. The authors' own prior-family control (Fig. 5d) shows that share recovered depends on whose prior generated the structure, which limits how far "42% vs. 23%" (Table 2) can be read as a property of the learners rather than of the estimator.

  4. The central result against BART is statistically fragile and concentrated. On the 36 spare-row tables the TFMs lose more than BART on only 23 (p = 0.132), five tables carry 84% of the summed gap, and without them the mean gap is 0.006 (L334-L336). The headline 37 of 52 (p = 0.003) is reached by pooling with the 16 CTR23 tables (Table 2). At larger contexts single tables dominate the means (L2019-L2027, L2041-L2045), so normalized MSE on heavy-tailed targets is driving several of the reported numbers. Code is promised only upon publication (L505-L507), so these choices cannot be checked at review time.

Questions for authors
  1. Scope of the claim. Given Table 22 and L963-L967, do you consider the claim that TFMs find more structure to hold beyond roughly 500 context rows? Please state in the abstract and Sec. 1 the regime (nn, dd, regression) in which the results are established. Relatedly, Table 15 shows several tables truncated from 32 to 122 raw features to 20 by column-pivoted QR (L1456-L1457): does the ablation gap change on the full-width versions of us_crime, ames_housing and CD4 under the native pipelines of App. I?

  2. Dataset universe. All 57 roster tables and the 16-table replication set are from OpenML (L282-L288). Have you run the audit on any tables from an independent source, and if not, could you add a set from outside OpenML? This would also address contamination: for the 23 non-screened tables (L282-L284), does the ablation gap differ from that on the 34 screened tables, and for TabDPT-Turbo can you confirm table-by-table exclusion rather than relying on the corpus rule (L285-L286)?

  3. Classification. Since exact twins do not exist for 0/1 targets (L485-L486), have you considered approximate twins, for example conditioning on a logistic sufficient statistic under a generalized linear null as in Janková et al. (2020), which you cite at L456-L457? Even an approximate audit on a few classification tables would address the largest gap in scope.

  4. Estimator dependence of the ablation. Could you report share recovered with at least one non-tree f^\hat{f} (GP, MLP or GAM) on the 36 spare-row tables, alongside the tree and TabPFN 3 estimates in Table 2? Fig. 5d suggests the number is prior-dependent; how should a reader interpret "42% vs. 23%" in light of that?

  5. Concentration of the BART result. With the five tables that carry 84% of the summed gap removed, the mean gap is 0.006 (L336). Is the interval for that reduced mean above zero? Please add it, and consider reporting Hodges-Lehmann or median gaps in Table 2 rather than only in Table 26.

  6. Code release. Would you be able to provide an anonymized repository during the discussion period, given that the results depend on several estimator choices (reference basis, screening rule, f^\hat{f}, KK, number of permutations)?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer sb9b, part 1 of 3: summary and context size (Q1)

Thank you for the detailed review. Your main point is that the evidence base was too narrow for a general claim. We agree, and did two things.

  • We extended the study: the full split (twins, ablation, noise redraws) at 500–10,000 context rows, wide and full-width tables, classification, 61 new OpenML tables, 16 tables from outside OpenML and non-tree structure estimators. We also release the code.
  • We narrowed the claim to what those runs support. The TFMs' accuracy advantage and the boosters' extra noise cost hold on new tables, inside and outside OpenML, and up to 2,000 context rows. The structure edge is a small-table result: it sits on the tables with the most structure, is larger over the boosters on wide tables at 120 rows, and is not consistently detected from 500 rows on. The submitted Limitations already reported the decline with context size; the abstract and Section 1 now state the regime.
Your questionShort answer
Q1. Beyond 500 rows?Not consistently. Over the boosters, the advantage and the noise edge persist to 2,000 rows and are small by 5,000–10,000. The structure edge is small throughout.
Q1. Full width?The gap is positive on all three tables against every baseline (two of three with native pipelines), and larger on wide tables at 120 rows (0.069, 15/18).
Q2. Other sources, contamination?61 new OpenML tables and 16 from outside OpenML (CARTE). The noise result replicates on both, and the structure gap sits on high-structure tables. No difference is detected for unscreened tables or for TabDPT's training-list tables.
Q3. Classification?Run on 37 binary tables. The gap over BART is 0.032 (31/37 tables). Over the boosters it is positive in the mean but not table by table.
Q4. A non-tree estimator?The mean gap over BART is positive with all seven estimators, and significant table by table for three.
Q5. Concentration of the BART result?Without the top five tables the interval includes zero: 0.006 [−0.002, 0.015]. The claim is narrowed.
Q6. Code?In the supplementary material.

Below, the primary measure is the paper's ablation gap in normalized MSE: the mean over tables with a percentile bootstrap interval, plus a sign test over tables. (Table 2 uses BCa intervals, so the same mean can show a slightly different interval.) All new runs are in Appendix O of the revised PDF.

Q1. Context size. We ran the full split at 500, 1,000 and 2,000 rows on 31, 28 and 24 tables and, in a pre-registered run with the structure estimated on up to four times as many rows as the context, at 5,000 and 10,000 rows on 20 and 16 tables (Appendix O.6):

BaselineContextAdvantageAblation gap (primary)Extra noise cost
CV-tuned boosting5000.040, 30/310.007 [−0.009, 0.021], 21/31, p = 0.070.010, 29/31
1,0000.027, 23/28−0.001 [−0.012, 0.008], 13/280.008, 26/28
2,0000.059, 22/240.045 [0.005, 0.116], 16/24, p = 0.150.007, 23/24
5,0000.019, 16/200.009 [−0.004, 0.022], 14/20, p = 0.120.003, 16/20
10,0000.022, 11/160.001 [−0.017, 0.017], 9/160.000, 10/16
BART5000.041, 27/310.011 [−0.002, 0.025], 18/310.007, 25/31
1,0000.035, 24/280.012 [0.000, 0.025], 20/28, p = 0.0360.011, 22/28
2,0000.054, 23/240.034 [0.012, 0.065], 16/24, p = 0.150.008, 20/24
5,0000.032, 16/200.019 [0.003, 0.038], 12/200.004, 17/20
10,0000.051, 12/160.023 [0.008, 0.042], 11/160.004, 15/16

From 500 to 2,000 rows, the TFMs still win on accuracy and still fit less noise than the boosters. Their structure edge is small, and the primary measure no longer detects it consistently against either baseline. At 2,000 rows one table (methane_emissions_rice_crop, gap 0.79 over boosting) carries the means. Without it (a post hoc exclusion) the gap is 0.013 [0.003, 0.025] over boosting and 0.021 [0.009, 0.033] over BART. The pre-registered rerun at 2,000 rows with that stronger estimate (28 tables) gives the same picture.

At 5,000–10,000 rows the advantage over the boosters is itself small (0.019 and 0.022 over CV-tuned boosting, about zero over LightGBM, as you noted from our appendix), and their extra noise cost is at most 0.003. So the two parts of the small-table advantage fade at different rates: the structure edge by about 500 rows, the noise edge by 5,000–10,000. Against BART the mean gap stays positive but not table by table. The structure claim is now stated for 120 rows.

Table 22. Its 0.000 at 1,000 rows (interval [−0.087, 0.085] in Table 24) came from one partition per table, and one table dominates that mean. The new run averages three partitions on 28 tables and gives a net-discovery gap over boosting of 0.015 [0.004, 0.027]. That lies inside the old interval, and the medians agree (0.005 and 0.008).

AuthorsOpenReview ↗

Reply to Reviewer sb9b, part 2 of 3: width and data sources (Q1, Q2)

Q1. Width.

  • Wide tables at 120 rows (pre-registered; Appendix O.7). Every eligible OpenML regression table with 50–500 features (40 tables), with the learners seeing up to 100 columns. 18 tables and 77 partitions remain after the paper's rules.
    • Over CV-tuned boosting the gap is 0.069 [0.020, 0.140], positive on 15/18 (p = 0.008, Holm 0.015). That is larger than with at most 20 features.
    • Over BART it is 0.071 [0.024, 0.137], positive on 13/18 (p = 0.096).
    • Two of the 18 tables are uploads of the same UCI data, which is also among our 57 tables. Without them the results are 13/16 over boosting (p = 0.021) and 11/16 over BART.
  • Wide tables at 1,000 rows (16 tables, 54 partitions). The TFMs come out ahead on 12–13 of 16 tables, but their mean advantage over the boosters is about zero, and the ablation gap is not detected.
  • Full-width us_crime, ames_housing and CD4 (27 partitions; the learners see every column). The TFMs win, and the ablation gap is positive, on all three tables against every baseline: 0.041–0.060 over BART and 0.011–0.081 over CV-tuned boosting. On us_crime that is larger than at 20 columns (0.010 and −0.038). With the TFMs in their native pipelines, the gap is positive on two of three (slightly negative on us_crime), and their accuracy advantage is mixed (Appendix O.8).

So at 120 rows the claim extends to wider tables against the boosters. Against BART it holds in the mean but not table by table. One caveat for the wide and full-width runs alike: the reference basis keeps 20 columns, so the removed structure includes linear signal in the extra columns.

Q2. Dataset universe and contamination.

  • New OpenML tables. Active OpenML regression tasks with 1,304–1,000,000 rows and 4–50 features, outside our 57 tables, OpenML-CTR23 and the wide-table set: 136 tables, shuffled with seed 20261114 and run at 120 rows in two batches (the first 60, then the rest). Sixteen are uploads of data already counted (eleven of them copies of tables among our 57), which name and size filters miss. We count each dataset once. After the paper's rules, 61 tables and 220 partitions remain (Appendix O.12).

    • Over CV-tuned boosting the gap is 0.034 [0.005, 0.079], or 0.015 [0.002, 0.027] without one table on which the boosting baseline fails. It is positive on 37/61 tables (p = 0.12). So it is positive on average, but not on a clear majority of tables.
    • Over BART it is 0.016 [−0.003, 0.035], positive on 34/61 (p = 0.44).
    • The gap sits where the structure is. It rises with the worth of the structure estimate against every baseline (rank correlation 0.24–0.50). On the two thirds of tables with the least worth it is near zero, because the ablation has little to remove. On the top third it is positive on 16–18 of 21 tables against each of five baselines (p ≤ 0.027).
    • The TFMs' accuracy advantage holds against every baseline (45–54/61), as does the boosters' extra noise cost (59/61 and 57/61).

    So the concentration you raised in W4 for BART holds on new tables, and against the boosters too: the TFMs find more structure where there is structure to find. These tables are again from OpenML, so they speak to fragility (W4), not to the single source (W2).

  • Outside OpenML (pre-registered). The CARTE benchmark's regression tables come mostly from Kaggle and other public sites. A fixed rule leaves out the 12 tables we found on OpenML by name or row count. After the paper's rules, 16 tables and 56 partitions remain (Appendix O.12).

    • The TFMs' advantage holds (14/16 over CV-tuned boosting, 12/16 over BART). The boosters' extra noise cost is positive on 15–16/16.
    • The structure gap is small: over BART 0.013 [0.004, 0.027], 12/16 (p = 0.077); over boosting 0.007 [−0.003, 0.018], 11/16 (p = 0.21). Neither sign test is significant (Holm 0.15 and 0.21). The structure estimate is worth much less here (0.075, against 0.22 on our tables), so this matches the pattern on the new OpenML tables.
    • A separate rule over UCI (not pre-registered) adds only two tables, with the same picture.
  • Screened vs unscreened (Appendix O.13). Among the 36 spare-row tables, the gap over BART is 0.027 on the 25 drawn tables and 0.046 on the 11 widely used ones (difference −0.018 [−0.085, 0.034]). Over boosting it is 0.031 vs 0.026. This is underpowered: no difference is detected, and we claim no more than that.

  • TabDPT-Turbo, table by table. TabDPT's public training code lists 130 OpenML datasets, and six of our 36 spare-row tables appear there. On those six, TabDPT's accuracy relative to the other three main TFMs is if anything lower than elsewhere (−0.017 [−0.046, 0.008]), and its ablation rise is no different (0.007 [−0.012, 0.032]). Without the six, the four TFMs' gap is 0.036 [0.009, 0.068] over BART and 0.035 [0.016, 0.056] over boosting (25/30 tables).

AuthorsOpenReview ↗

Reply to Reviewer sb9b, part 3 of 3: classification, estimator, BART and code (Q3–Q6)

Q3. Classification. We ran the ablation and noise redraws on 37 binary tables (OpenML-CC18, the AutoML benchmark and TabArena; 120 rows; 85 partitions; Appendix O.9). The TFMs here are the three with freely available classifiers (TabICLv2, TabDPT, TabPFN v2). To remove the structure beyond the linear part of the logit, we couple labels to uniforms: U_i ~ U(0, p̂_i) if y_i = 1, else U(p̂_i, 1). Then 1[U < p̂] reproduces the real labels, and 1[U < expit(linear part)] flips only the labels that must change (8% on average). The loss is Brier divided by the label variance.

BaselineAdvantageAblation gapExtra noise cost
BART0.042 [0.018, 0.072], 26/370.032 [0.014, 0.057], 31/37, p < 0.001−0.028 (BART fits less)
CV-tuned boosting0.049, 31/370.016 [0.002, 0.030], 20/37, p = 0.740.016, 30/37
Optuna LightGBM0.047, 31/370.020 [0.008, 0.035], 23/37, p = 0.190.013, 29/37

Against BART the advantage is mostly structure. Against the boosters the structure gap is positive in the mean but not table by table (20/37, 23/37), while the noise gap is consistent (30/37, 29/37). The abstract makes no classification claim. Limitations: we did not build the approximate twins you suggest (Janková et al., 2020), so there is no ΔB, and the label coupling has no planted-truth check yet.

Q4. A non-tree estimator, and how to read "42% vs. 23%". On the 36 spare-row tables (275 partitions), with the structure also estimated by six other methods, each screened on its own audit rows (Appendix O.2; percentile intervals):

EstimatorShare, TFMs vs BARTGap over BARTSign test
Trees (paper)36% vs 21%0.033 [0.010, 0.061]23/36, p = 0.13
TabPFN 345% vs 27%0.032 [0.012, 0.059]26/36, p = 0.011
GP68% vs 45%0.033 [0.011, 0.065]23/31, p = 0.011
MLP55% vs 34%0.031 [0.011, 0.059]26/34, p = 0.003
k-NN69% vs 44%0.025 [0.005, 0.054]19/29, p = 0.14
GAM (additive)72% vs 51%0.012 [0.005, 0.019]20/35, p = 0.50
Average of four51% vs 32%0.030 [0.011, 0.056]22/35, p = 0.18

The mean gap is positive with every estimator, and significant table by table for three of seven. Share levels depend on the estimator, since a share is relative to what the estimator captures, but the TFMs' share is 1.4–1.7 times BART's with every estimator. So "42% vs. 23%" means "of what a tree ensemble finds on spare rows", and Table 2's caption now says so. Fig. 5d adds that the share also depends on the kind of structure (BART and the GP recover more on their own prior families), so it describes these tables, not the learners in general. (The TabPFN 3 row differs slightly from Table 2's, 44% vs 26% and 0.035, because each estimate is now screened on its own audit rows.)

On the shared inductive bias: a tree-based estimate should, if anything, favor the tree-based boosters. Yet at 120 rows the TFMs' forecasts correlate more with the estimated structure than every baseline's (29–32 of 36 tables; by the regression coefficient on it, 25/36 against CV-tuned boosting and LightGBM), and over the boosters the same holds with the MLP estimate (reply to gy5u).

Q5. Concentration of the BART result. Without the five tables that carry most of the summed gap, the mean is 0.006 [−0.002, 0.015] (0.007 in the rerun that supplies the interval). This interval is not above zero, and Section 4.1 now prints it. The Hodges–Lehmann estimate over all 36 tables is 0.015 [0.002, 0.029] (Appendix O.5; Table 2's caption now gives it). The pooled 37 of 52 (p = 0.003) stays in the abstract with its two sets named; Table 2 gives them separately (23/36, p = 0.132; 14/16, p = 0.004), and the claim drawn from them is the narrowed one.

Following your remark that heavy-tailed targets drive several numbers, we also computed the gap under losses that a few extreme targets cannot dominate. This is a post hoc robustness check, labeled as such in the paper and not in the abstract. In absolute error the gap over BART is 0.033 [0.019, 0.049], positive on 29/36 tables (p = 0.0003), and 0.019 [0.011, 0.027] without the top five. With MSE clipped to the 1st–99th percentile it is positive on 25/36 (p = 0.029).

So in MSE the BART result is concentrated, as the submitted Section 4.1 reported and you stressed. The 61 new tables (Q2) show the same: no table-by-table gap over BART in MSE (34/61), and a gap on the tables with the most structure (17/21). We have therefore narrowed the BART claim in the abstract, rather than relying on the post hoc loss.

Q6. Code. The anonymized code is in the supplementary material of the revision: the drivers, configurations and analysis scripts. It covers the reference basis, the screening rule, f̂, K and the number and seeds of the noise permutations.

I'm retaining my assessment. The claim requires more study and while the responses address some issue but isn't addressing all the queries raised.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. In short:

  • What holds. The TFMs' accuracy advantage, and CV-tuned boosting's and LightGBM's extra cost of fitting noise. They hold on 45–54/61 and 57–59/61 of the new OpenML tables, on 16 tables from outside OpenML (14/16 and 15/16 against CV-tuned boosting), and at 500 to 2,000 context rows.
  • What we narrowed. The TFMs find more structure on small tables that have much of it. On the new tables the ablation gap is positive on 16–18/21 of the high-structure third against each of five baselines, and near zero on the rest. From 500 rows on it is not consistently detected. By 5,000–10,000 rows the advantage over the boosters is itself small, as the submitted paper reported against LightGBM.
  • What is new. Checks that the sign of the ablation gap reflects discovery (gy5u), a factorial split after eLKi's proposal (Section 2.4), and the anonymized code, in the supplementary material.

All new runs are in a new Appendix O, and every run is reported. The primary measure is the paper's ablation gap in normalized MSE: the mean over tables with a percentile bootstrap interval, and a sign test over tables. Other losses and summaries that drop a single heavy-tailed table are secondary and labeled as such; the absolute-error check at 120 rows is post hoc.

QuestionWhat we ran and found
Is the ablation gap discovery, or adaptation and noise fitting? (gy5u)We split each learner's rise into the part along the removed structure and the rest. The TFMs' extra rise lies along the structure: 0.034 over BART, 0.021 over CV-tuned boosting and 0.019 over LightGBM (27–30/36 tables). Freezing the baselines' pipelines changes their rise by at most 0.005. Without noise and with pipelines frozen, the gap is larger: 0.040 over BART (30/36) and 0.048 over boosting (35/36). With no label changed, the TFMs' forecasts correlate more with the structure than every baseline's (29–32/36).
Does it depend on the structure estimator? (sb9b, gy5u)With trees, a GP, MLP, GAM, k-NN, TabPFN 3 and an average, the mean gap over BART is positive every time. Table by table it is significant for three of seven. It shrinks as the estimate weakens.
Does the result rest on a few tables? (sb9b, gy5u)Partly yes. On 61 new OpenML tables the gap sits where the structure is (see above). Over all 61 tables the gap over CV-tuned boosting is positive in the mean (0.034 [0.005, 0.079]; 0.015 [0.002, 0.027] without one table) but on only 37/61 tables. Over BART it is 0.016 [−0.003, 0.035], on 34/61.
How does the split change with context size? (gy5u, sb9b)At 500, 1,000 and 2,000 rows boosting catches up on structure and on the linear fit. Its extra noise cost stays positive on 29/31, 26/28 and 23/24 tables. At 5,000 and 10,000 rows that cost is at most 0.003, and the advantage over the boosters is small.
Wider tables, classification? (sb9b)At 120 rows on 18 wide tables (pre-registered), the gap over CV-tuned boosting is larger than with at most 20 features: 0.069, 15/18 tables (Holm p = 0.015); over BART 13/18 (p = 0.096). At 1,000 rows on 16 wide tables it is not detected. On full-width us_crime, ames_housing and CD4 it is positive on all three tables against every baseline (two of three with native TFM pipelines). In classification (37 binary tables) the gap over BART is positive on 31/37 tables; over the boosters it is positive in the mean but not table by table.
Tables from another source, contamination (sb9b)On 16 tables from outside OpenML (CARTE benchmark; pre-registered), the boosters' extra noise cost is positive on 15–16/16. The ablation gap is small: 12/16 over BART (p = 0.077) and 11/16 over boosting. TabDPT does not stand out on the six tables in its training list. Drawn and widely used tables do not differ detectably (wide intervals).
Simpler design, fairer baselines (eLKi)The factorial split agrees with Eq. 6 on the structure part, and shows that much of ΔB against the boosters is an effect of the residual noise. Against 5×5 repeated-CV and bagged boosting, repeated-CV BART, a wider-grid BART, and boosters with native categorical handling, the TFMs still win on 28–30/36 tables (native boosters: the mean advantage's interval includes zero).

What changes in the paper

  1. Scope. The abstract and Section 1 state the regime: regression, 120 context rows and at most 20 features for the structure claims. With the Limitations, they also say what larger contexts, the 61 new tables and the tables from outside OpenML show.
  2. Where the structure gap holds. The submitted paper already placed the largest gaps over BART on the tables with the most structure. The abstract now says the same for the boosters on the new tables.
  3. Terms and language. The terms eLKi flagged are replaced, forecasts are written ŷ, and the paper is rewritten in plainer language.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.