- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks whether tabular foundation models (TFMs) can lose an interaction when other columns in the table become useful. The authors add a known three-way parity interaction to real regression targets. They then compare models given the original columns with models given permuted copies of those columns. This keeps the target, table width and column distributions the same. That separates the effect they call “crowding out” from “dilution”, the loss caused by adding uninformative columns. TabPFN-3, TabICLv2 and TabFM lose part of the interaction when the other columns are informative. Boosting models and most gradient-trained networks gain from the same columns. Changing the labels the TFMs use to build column representations, while keeping prediction labels fixed, turns the effect on and off.
The paper also introduces CrowdBench, which tests interaction retention on controlled tasks, datasets with known rules, and measured systems such as antibody binding and GPU runtimes. In some measured cases, accuracy improves while part of an interaction is lost; in others, added summary features hurt accuracy.
Critical strengths
- A clean experimental design that separates crowding from dilution.
The paper compares models given the original informative columns v/s models given permuted copies of the same columns. The target, table width, and column distributions remain the same, so the experiment isolates the effect of informative columns on interaction capture. With 16,000 context rows, TabPFN-3 captures 0.76 of the planted parity interaction with permuted columns but only 0.35 with aligned columns, as shown in figure 1. At full width, TabICLv2 and TabFM already struggle to learn the parity, even with permuted columns. However, when the tables are narrowed to two or four real columns, all three TFMs show clear crowding, averaging 0.30–0.34, while CatBoost shows a small effect in the opposite direction (−0.06; Sec. 3.2, Appendix B). The effect also appears on 22 additional tables, in binary classification, under joint permutation of competing columns, and with TabPFN-2.5. In contrast, boosting and gradient-trained networks generally benefit from the same informative columns. These results suggest that the effect depends on how the learner uses the available information rather than being caused by the task alone. The matched experimental design is a key methodological contribution that could be reused to study other models.
- Intervention experiments provide strong evidence about where the effect enters the model.
The column-label interventions are one of the most convincing experiments of the paper. The authors change the labels used to construct column representations while keeping the labels used for prediction fixed. Removing the competing signal from the column labels raises interaction capture from 0.08 to 0.65 for TabICLv2, from 0.33 to 0.78 for TabFM, and from 0.18 to 0.60 for TabPFN-3. Reversing the intervention induces the suppression again. Control perturbations with the same variance change capture by at most 0.05, suggesting that the effect depends on the information in the labels rather than simply their variation. Further training of TabICLv2 on its original prior reduces the suppression gap by about 60% in classification, showing that the effect is not an unavoidable property of the architecture. These experiments show that the effect exists and how it may arise as well.
Critical weaknesses
- The practical importance of crowding on real-world targets remains unclear.
The clearest results use a planted pure parity interaction. This interaction has no signal in any individual column or smaller group of columns, and in the main benchmark it accounts for about a quarter of the target variance. Crowding is often much smaller when the interaction also has simpler patterns, which is common in real data. The paper's own analysis also suggests that pure higher-order structure is a small part of many real targets: across 20 protein datasets, main effects and pairwise interactions explain a median of 96% of the variance. Figure 3 shows that crowding can persist when a small additive clue is added to a fixed parity, so the effect does not require a target with no simpler signal at all. However, the evidence of meaningful harm on measured data remains limited:
- The pure third-order components missed by TFMs explain a median of only 0.1–0.3% of residual variance.
- On CH65, the loss in interaction capture occurs alongside a 12 to 15% improvement in RMSE. TabM also loses more interaction capture than any of the TFMs.
- On SGEMM, the RMSE increases do not pass the paper's own test for harm above 3%.
- The FPS result was selected after screening 31 families, and across all 31 families, adding summaries improves average TFM RMSE by 1.9-2.9%.
- The large performance loss on KTT also affects TabM, with a 38.2% RMSE increase on GEMM/2080.
There is also an important limitation at full table width. TabICLv2 and TabFM already struggle to learn the parity when the competing columns are permuted. For these models, the effect becomes much clearer only when the table is narrowed to two or four real columns. Thus, at full width, dilution appears to be a more important limitation than crowding for these two models. Overall, the controlled experiments establish that crowding is a real phenomenon, but they do not establish how often it causes meaningful harm on ordinary datasets. The authors themselves describe the measured results as evidence of susceptibility rather than prevalence. Since practical relevance is a central motivation of the paper, this would be my main concern.
- The paper covers too many contributions, and several secondary contributions are not sufficiently supported.
Beyond the main finding and the column-label interventions, the paper includes readout-transfer experiments, feature-allocation studies involving forests and xRFM, continued training, a three-track benchmark, a residual-search algorithm with theoretical guarantees, and a significance-based guard for applying corrections. Including all these contributions makes the paper difficult to follow, and several of the secondary results have important limitations:
- Proposition 2 assumes that columns are scored individually, but the authors acknowledge that TFMs do not follow this behaviour in the appendix.
- Proposition 4 does not cover the pooled cross-fitting procedure used in the experiments.
- The residual-search method depends on its search dictionary and fails to recover any of the 10 tested supports for an all-even interaction.
- Ridge regression on a single discovered product matches or outperforms the TFM-based correction.
- The authors note that gains on native tables “mostly come from generic residual stacking”
These issues do not change the main result. However, the additional contributions increase the paper's length without providing equally strong support for its central claims. A shorter paper focused on the crowding phenomenon, the column-label mechanism, and stronger evidence of practical relevance would present a clearer and more convincing contribution.
Questions for authors
-
How often does crowding occur on ordinary datasets without planted interactions? Could you apply the aligned-versus-permuted probe to the native regression families or to datasets from TabArena? Please report how often significant crowding of an unplanted component is detected and the size of the resulting RMSE change.
-
Is crowding a model limitation or a consequence of its pretraining prior? If the synthetic prior favours targets that depend on only a few columns, a model that follows this prior closely might reasonably focus on an informative signal rather than a pure parity interaction. Could you test crowding on tasks sampled from each model's own pretraining prior, or compare against a Bayesian learner with a prior favoring a small number of relevant columns?
-
What connects the measured harms to the proposed column-label mechanism? The controlled label-swap experiments show that changing column labels can induce or reduce crowding. However, the measured results are less conclusive about whether the same mechanism explains the observed harms. On CH65, TabM loses more interaction capture than any of the TFMs. On SGEMM, replacing column labels with residual labels makes predictions worse, which differs from the improvement observed in the controlled experiments. What evidence connects these measured harms to the proposed column-label mechanism?
5 comments
Response to Reviewer MCP9 (2/2)
5. Model limitation or pretraining prior
“Is crowding a model limitation or a consequence of its pretraining prior? ... Could you test crowding on tasks sampled from each model's own pretraining prior, or compare against a Bayesian learner with a prior favoring a small number of relevant columns?”
We thank the reviewer for suggesting the Bayesian comparison, which made the hypothesis precise. We ran it, along with the closest prior test that public generators allow; the question remains unresolved, as Section 7 says. The prior test covers only TabICLv2, because the generators of TabPFN-3 and TabFM are not public and, for TabICLv2, only its predecessor's is.
- Competitors from TabICL's public (v1) prior. We drew the competing table from that generator, 40 draws per context size, and planted the parity on top, so the target itself is not a prior draw. Where the parity is learnable in a separate selection fit, TabICLv2 crowds by 0.37 [0.29, 0.45] with 1,024 rows and 0.30 [0.21, 0.40] with 4,096. Over all 40 draws it crowds by 0.09 [0.05, 0.15] and 0.16 [0.10, 0.22]. CatBoost benefits.
- An exact Bayesian learner. Under independent-inclusion or beta-binomial sparsity priors it does not crowd (−0.04 to +0.01). Under a hard budget of four relevant columns, fewer than the true model needs, it crowds in every cell (0.41–1.02), largely by construction. Post hoc, a prior against the product term never crowds. So a preference for few relevant columns does not by itself predict crowding; a budget does.
- Continued training. Further training on the v1 generator removes 91% of the classification suppression gap, from 0.67 to 0.06. Our Table 7 called this absolute change (0.60 and 0.62) only a “reduction”, which reads as a percentage; the row is now labeled absolute. The training costs reference capture and ordinary-table log loss, and the gain and its costs may partly reflect the change of prior.
Revised manuscript: Section 7; Appendices G and G.1; Tables 7, 9 and 10.
6. Measured harms and the column-label mechanism
“What evidence connects these measured harms to the proposed column-label mechanism?”
For the harms in the submission, little; for TabICLv2 on the two unplanted targets, the swap works. In each case we replace only the labels seen by the column or cell encoder; inputs and prediction labels stay bit-identical.
- CH65. We replaced the column-stage labels with the target minus the summary. As prespecified, this beats a matched random direction for TabICLv2 (+0.08 [0.04, 0.12]) and TabFM (+0.16 [0.12, 0.20]), but not for TabPFN-3. Against intact labels, however, capture changes by only −0.01, −0.09 and +0.02, so little of the loss is released (16% of the gap for TabFM, none for the others). TabM, which has no column-stage labels, loses even more capture there. We therefore do not attribute the summary-induced harms to the pathway.
- SGEMM. As you note, residual column labels make predictions worse: beyond-pair MSE rises in every arm, by 17.5% for TabPFN-3 and 27.2% for TabICLv2 with aligned inputs. We therefore do not attribute the runtime harms to the pathway either.
- Weather, TabICLv2. Labels built from the context rows alone raise capture from 0.62 to 0.87 with 4,000 rows and from 0.85 to 0.94 with 16,000, and lower RMSE by 3.7% and 1.8%; we added them after registration. The registered labels subtract station-month means. Those means use every row of the station block except the draw's test rows, so they draw on labels outside the context; they raise capture to 0.77 and 0.91. A registered linear version beats its matched direction but not intact labels. With the covariates permuted, none of these swaps raises capture (post hoc).
- Weather, TabPFN-3. Context-only labels raise capture from 0.48 to 0.74 with 1,000 rows and from 0.81 to 0.86 with 2,000, but RMSE does not improve: it changes by −0.9% [−4.6, 2.9] and +2.7%.
- Food delivery, TabICLv2 (registered, new draws). Removing the age-and-rating fit from the column-stage labels releases about half of the loss (+0.34 [0.27, 0.41]) and lowers RMSE by 2.0% (descriptive).
The matched random directions are not neutral on the weather table, so Table 31 also reports each gain over intact labels.
Revised manuscript: Sections 5 and 7; Appendices H, I, P and Q; Tables 12, 13 and 27–33.
We hope these revisions address your concerns, and we would be very glad to clarify any remaining point.
Thank you for the detailed response and the new experiments. The weather table, the TabArena components and the Bayesian comparison mostly answer my questions, and the paper is now clear about what its results do not show. However, my main concern was whether crowding matters in practice, and the new results do not change that. On ordinary tables, crowding is rarely found and costs at most about 1% of test MSE. I will hold my rating.
Response to Reviewer MCP9 (1/2)
We sincerely thank the reviewer for the expert and constructive review, and for recognizing the matched design and the column-label interventions. We agree that the paper does not establish prevalence, and it now claims less: a loss that RMSE can hide, a matched test for it, a causal handle that also works on two unplanted targets (the second chosen after a first look), and evidence that the loss also arises with competitors from TabICL's public prior. The shared new evidence is in the general response (G1–G5). We addressed each question and weakness as follows.
1. Crowding without planted interactions
“How often does crowding occur on ordinary datasets without planted interactions? ... Please report how often significant crowding of an unplanted component is detected and the size of the resulting RMSE change.”
On ordinary tables, rarely and at little cost. On TabArena, 3 of 45 TFM contrasts stay positive after Holm adjustment and a post hoc rule that the sign must hold when predictions are regressed on the component alone; they cost 0.1–0.8% of test MSE (G2). Among 13 components that a calibrated test shows the TFMs leave unused, we detect none (G3). It does occur on a weather table that the mechanism singles out (G1), where releasing the loss through context-only column-stage labels lowers TabICLv2's RMSE by 3.7% with 4,000 rows.
Revised manuscript: Section 5; Appendices L, P and Q; Tables 31–33.
2. Full width and variance share
“Thus, at full width, dilution appears to be a more important limitation than crowding for these two models.”
We agree, and Section 3.2 now says so. The crowded weather interaction holds about 4% of the variance, and the food-delivery component 1.3%. In a planted pilot (Table 25), TabICLv2 and TabFM crowd cyclic sums and matches of two categorical columns (0.30–0.66), and TabPFN-3 two of six shapes. Post hoc, on the 13 TabArena tasks each TFM beats CatBoost in RMSE on 12–13 tasks with nothing planted, and on at most one once the parity, about a quarter of the target variance, is planted; that reversal combines crowding and dilution.
Revised manuscript: Section 3.2; Appendices O and P; Tables 24 and 25.
3. Measured harms and selection
“However, the evidence of meaningful harm on measured data remains limited.”
We agree with the points behind this: RMSE improves on CH65, SGEMM fails our own 3% test, FPS was selected from 31 families, and TabM shares the KTT harm. Section 5 now states the selection beside each result. Across all 31 families, full summaries improve average RMSE by 1.9% for TabPFN-3 and 2.9% for TabICLv2 (the screen's two TFMs), while grouped summaries, as in FPS, cost TabPFN-3 1.9%.
Revised manuscript: Section 5; Table 2; Appendix I.
4. Scope and length
“A shorter paper focused on the crowding phenomenon, the column-label mechanism, and stronger evidence of practical relevance would present a clearer and more convincing contribution.”
We followed this in part: the main text is refocused, but the paper is longer.
Revised manuscript: Sections 5 and 6; Appendices K, K.1 and M.