acceptodds
Under review as a conference paper at ICLR 2027

Useful Features Can Crowd Out Interactions in Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) can lose an interaction when other columns become informative, even though those columns remain useful at test time. We call this crowding out. The TFMs predict from labeled context rows with fixed pretrained weights. We add a known parity component to real regression targets and compare the original competing columns with permuted copies. The target, table width and column marginals stay fixed, separating crowding from dilution by uninformative columns. TabPFN-3, TabICLv2 and TabFM exhibit crowding; boosting and most networks trained by gradient descent benefit from the same columns. Weak additive or pairwise clues improve interaction learning without preventing crowding. Swapping labels used to construct column representations induces or releases crowding while prediction labels stay fixed. CrowdBench tests interaction retention on controlled tasks, known rules and measured systems. In selected measured systems, summaries improve RMSE while hiding component loss, or hurt prediction as context grows. These cases establish susceptibility rather than prevalence. Searching prediction residuals and refitting on the discovered columns restores planted interactions within the search dictionary.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular data, Tabular foundation models, feature interactions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MCP9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether tabular foundation models (TFMs) can lose an interaction when other columns in the table become useful. The authors add a known three-way parity interaction to real regression targets. They then compare models given the original columns with models given permuted copies of those columns. This keeps the target, table width and column distributions the same. That separates the effect they call “crowding out” from “dilution”, the loss caused by adding uninformative columns. TabPFN-3, TabICLv2 and TabFM lose part of the interaction when the other columns are informative. Boosting models and most gradient-trained networks gain from the same columns. Changing the labels the TFMs use to build column representations, while keeping prediction labels fixed, turns the effect on and off.

The paper also introduces CrowdBench, which tests interaction retention on controlled tasks, datasets with known rules, and measured systems such as antibody binding and GPU runtimes. In some measured cases, accuracy improves while part of an interaction is lost; in others, added summary features hurt accuracy.

Critical strengths
  1. A clean experimental design that separates crowding from dilution.

The paper compares models given the original informative columns v/s models given permuted copies of the same columns. The target, table width, and column distributions remain the same, so the experiment isolates the effect of informative columns on interaction capture. With 16,000 context rows, TabPFN-3 captures 0.76 of the planted parity interaction with permuted columns but only 0.35 with aligned columns, as shown in figure 1. At full width, TabICLv2 and TabFM already struggle to learn the parity, even with permuted columns. However, when the tables are narrowed to two or four real columns, all three TFMs show clear crowding, averaging 0.30–0.34, while CatBoost shows a small effect in the opposite direction (−0.06; Sec. 3.2, Appendix B). The effect also appears on 22 additional tables, in binary classification, under joint permutation of competing columns, and with TabPFN-2.5. In contrast, boosting and gradient-trained networks generally benefit from the same informative columns. These results suggest that the effect depends on how the learner uses the available information rather than being caused by the task alone. The matched experimental design is a key methodological contribution that could be reused to study other models.

  1. Intervention experiments provide strong evidence about where the effect enters the model.

The column-label interventions are one of the most convincing experiments of the paper. The authors change the labels used to construct column representations while keeping the labels used for prediction fixed. Removing the competing signal from the column labels raises interaction capture from 0.08 to 0.65 for TabICLv2, from 0.33 to 0.78 for TabFM, and from 0.18 to 0.60 for TabPFN-3. Reversing the intervention induces the suppression again. Control perturbations with the same variance change capture by at most 0.05, suggesting that the effect depends on the information in the labels rather than simply their variation. Further training of TabICLv2 on its original prior reduces the suppression gap by about 60% in classification, showing that the effect is not an unavoidable property of the architecture. These experiments show that the effect exists and how it may arise as well.

Critical weaknesses
  1. The practical importance of crowding on real-world targets remains unclear.

The clearest results use a planted pure parity interaction. This interaction has no signal in any individual column or smaller group of columns, and in the main benchmark it accounts for about a quarter of the target variance. Crowding is often much smaller when the interaction also has simpler patterns, which is common in real data. The paper's own analysis also suggests that pure higher-order structure is a small part of many real targets: across 20 protein datasets, main effects and pairwise interactions explain a median of 96% of the variance. Figure 3 shows that crowding can persist when a small additive clue is added to a fixed parity, so the effect does not require a target with no simpler signal at all. However, the evidence of meaningful harm on measured data remains limited:

  • The pure third-order components missed by TFMs explain a median of only 0.1–0.3% of residual variance.
  • On CH65, the loss in interaction capture occurs alongside a 12 to 15% improvement in RMSE. TabM also loses more interaction capture than any of the TFMs.
  • On SGEMM, the RMSE increases do not pass the paper's own test for harm above 3%.
  • The FPS result was selected after screening 31 families, and across all 31 families, adding summaries improves average TFM RMSE by 1.9-2.9%.
  • The large performance loss on KTT also affects TabM, with a 38.2% RMSE increase on GEMM/2080.

There is also an important limitation at full table width. TabICLv2 and TabFM already struggle to learn the parity when the competing columns are permuted. For these models, the effect becomes much clearer only when the table is narrowed to two or four real columns. Thus, at full width, dilution appears to be a more important limitation than crowding for these two models. Overall, the controlled experiments establish that crowding is a real phenomenon, but they do not establish how often it causes meaningful harm on ordinary datasets. The authors themselves describe the measured results as evidence of susceptibility rather than prevalence. Since practical relevance is a central motivation of the paper, this would be my main concern.

  1. The paper covers too many contributions, and several secondary contributions are not sufficiently supported.

Beyond the main finding and the column-label interventions, the paper includes readout-transfer experiments, feature-allocation studies involving forests and xRFM, continued training, a three-track benchmark, a residual-search algorithm with theoretical guarantees, and a significance-based guard for applying corrections. Including all these contributions makes the paper difficult to follow, and several of the secondary results have important limitations:

  • Proposition 2 assumes that columns are scored individually, but the authors acknowledge that TFMs do not follow this behaviour in the appendix.
  • Proposition 4 does not cover the pooled cross-fitting procedure used in the experiments.
  • The residual-search method depends on its search dictionary and fails to recover any of the 10 tested supports for an all-even interaction.
  • Ridge regression on a single discovered product matches or outperforms the TFM-based correction.
  • The authors note that gains on native tables “mostly come from generic residual stacking”

These issues do not change the main result. However, the additional contributions increase the paper's length without providing equally strong support for its central claims. A shorter paper focused on the crowding phenomenon, the column-label mechanism, and stronger evidence of practical relevance would present a clearer and more convincing contribution.

Questions for authors
  1. How often does crowding occur on ordinary datasets without planted interactions? Could you apply the aligned-versus-permuted probe to the native regression families or to datasets from TabArena? Please report how often significant crowding of an unplanted component is detected and the size of the resulting RMSE change.

  2. Is crowding a model limitation or a consequence of its pretraining prior? If the synthetic prior favours targets that depend on only a few columns, a model that follows this prior closely might reasonably focus on an informative signal rather than a pure parity interaction. Could you test crowding on tasks sampled from each model's own pretraining prior, or compare against a Bayesian learner with a prior favoring a small number of relevant columns?

  3. What connects the measured harms to the proposed column-label mechanism? The controlled label-swap experiments show that changing column labels can induce or reduce crowding. However, the measured results are less conclusive about whether the same mechanism explains the observed harms. On CH65, TabM loses more interaction capture than any of the TFMs. On SGEMM, replacing column labels with residual labels makes predictions worse, which differs from the improvement observed in the controlled experiments. What evidence connects these measured harms to the proposed column-label mechanism?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MCP9 (1/2)

We sincerely thank the reviewer for the expert and constructive review, and for recognizing the matched design and the column-label interventions. We agree that the paper does not establish prevalence, and it now claims less: a loss that RMSE can hide, a matched test for it, a causal handle that also works on two unplanted targets (the second chosen after a first look), and evidence that the loss also arises with competitors from TabICL's public prior. The shared new evidence is in the general response (G1–G5). We addressed each question and weakness as follows.

1. Crowding without planted interactions

“How often does crowding occur on ordinary datasets without planted interactions? ... Please report how often significant crowding of an unplanted component is detected and the size of the resulting RMSE change.”

On ordinary tables, rarely and at little cost. On TabArena, 3 of 45 TFM contrasts stay positive after Holm adjustment and a post hoc rule that the sign must hold when predictions are regressed on the component alone; they cost 0.1–0.8% of test MSE (G2). Among 13 components that a calibrated test shows the TFMs leave unused, we detect none (G3). It does occur on a weather table that the mechanism singles out (G1), where releasing the loss through context-only column-stage labels lowers TabICLv2's RMSE by 3.7% with 4,000 rows.

  • Weather. We chose the covariates-only contrast during development. Post hoc, the all-columns contrast nets out two effects: against their permuted copies, place and season columns raise TabICLv2's capture by 0.10 [0.03, 0.16] with 4,000 rows and 0.09 [0.05, 0.12] with 16,000, while the weather covariates lower it by 0.11 and 0.07. Food delivery shows a similar split. We read both descriptively, not as a rule. With 1,000 rows or fewer the covariates help TabICLv2.
  • TabArena. Two of the three remaining contrasts are on a task whose permuted columns nearly duplicate one of the component's columns (correlations up to 0.94), so they may reflect substitution. Each table contributes one estimated, binned pair, so a null does not show that a table has no crowded interaction.

Revised manuscript: Section 5; Appendices L, P and Q; Tables 31–33.

2. Full width and variance share

“Thus, at full width, dilution appears to be a more important limitation than crowding for these two models.”

We agree, and Section 3.2 now says so. The crowded weather interaction holds about 4% of the variance, and the food-delivery component 1.3%. In a planted pilot (Table 25), TabICLv2 and TabFM crowd cyclic sums and matches of two categorical columns (0.30–0.66), and TabPFN-3 two of six shapes. Post hoc, on the 13 TabArena tasks each TFM beats CatBoost in RMSE on 12–13 tasks with nothing planted, and on at most one once the parity, about a quarter of the target variance, is planted; that reversal combines crowding and dilution.

Revised manuscript: Section 3.2; Appendices O and P; Tables 24 and 25.

3. Measured harms and selection

“However, the evidence of meaningful harm on measured data remains limited.”

We agree with the points behind this: RMSE improves on CH65, SGEMM fails our own 3% test, FPS was selected from 31 families, and TabM shares the KTT harm. Section 5 now states the selection beside each result. Across all 31 families, full summaries improve average RMSE by 1.9% for TabPFN-3 and 2.9% for TabICLv2 (the screen's two TFMs), while grouped summaries, as in FPS, cost TabPFN-3 1.9%.

Revised manuscript: Section 5; Table 2; Appendix I.

4. Scope and length

“A shorter paper focused on the crowding phenomenon, the column-label mechanism, and stronger evidence of practical relevance would present a clearer and more convincing contribution.”

We followed this in part: the main text is refocused, but the paper is longer.

  • Changed. Section 6 is a third shorter and presents recovery as a diagnostic; its figure moved to Appendix K. It claims no recovery on native tables, where gains mostly match generic stacking; puma32H stays as a case of dilution. In separate-sample controls, ridge on the discovered product matches the TFM refit, so finding the support, not the refit learner, drives recovery. The unplanted results have their own paragraph in Section 5.
  • Relabeled. Proposition 2 is a stylized allocation model of forests and xRFM, not of TFM computation. Proposition 4 names what it covers, the independent-sample protocol of Appendix K.1, and what it does not: the TFM refits and the pooled guard.
  • Kept at one paragraph each. The readout and allocation analyses, which two other reviews ask about.
  • Longer. The requested analyses add appendix pages, and the main text uses the tenth page.

Revised manuscript: Sections 5 and 6; Appendices K, K.1 and M.

AuthorsOpenReview ↗

Response to Reviewer MCP9 (2/2)

5. Model limitation or pretraining prior

“Is crowding a model limitation or a consequence of its pretraining prior? ... Could you test crowding on tasks sampled from each model's own pretraining prior, or compare against a Bayesian learner with a prior favoring a small number of relevant columns?”

We thank the reviewer for suggesting the Bayesian comparison, which made the hypothesis precise. We ran it, along with the closest prior test that public generators allow; the question remains unresolved, as Section 7 says. The prior test covers only TabICLv2, because the generators of TabPFN-3 and TabFM are not public and, for TabICLv2, only its predecessor's is.

  • Competitors from TabICL's public (v1) prior. We drew the competing table from that generator, 40 draws per context size, and planted the parity on top, so the target itself is not a prior draw. Where the parity is learnable in a separate selection fit, TabICLv2 crowds by 0.37 [0.29, 0.45] with 1,024 rows and 0.30 [0.21, 0.40] with 4,096. Over all 40 draws it crowds by 0.09 [0.05, 0.15] and 0.16 [0.10, 0.22]. CatBoost benefits.
  • An exact Bayesian learner. Under independent-inclusion or beta-binomial sparsity priors it does not crowd (−0.04 to +0.01). Under a hard budget of four relevant columns, fewer than the true model needs, it crowds in every cell (0.41–1.02), largely by construction. Post hoc, a prior against the product term never crowds. So a preference for few relevant columns does not by itself predict crowding; a budget does.
  • Continued training. Further training on the v1 generator removes 91% of the classification suppression gap, from 0.67 to 0.06. Our Table 7 called this absolute change (0.60 and 0.62) only a “reduction”, which reads as a percentage; the row is now labeled absolute. The training costs reference capture and ordinary-table log loss, and the gain and its costs may partly reflect the change of prior.

Revised manuscript: Section 7; Appendices G and G.1; Tables 7, 9 and 10.

6. Measured harms and the column-label mechanism

“What evidence connects these measured harms to the proposed column-label mechanism?”

For the harms in the submission, little; for TabICLv2 on the two unplanted targets, the swap works. In each case we replace only the labels seen by the column or cell encoder; inputs and prediction labels stay bit-identical.

  • CH65. We replaced the column-stage labels with the target minus the summary. As prespecified, this beats a matched random direction for TabICLv2 (+0.08 [0.04, 0.12]) and TabFM (+0.16 [0.12, 0.20]), but not for TabPFN-3. Against intact labels, however, capture changes by only −0.01, −0.09 and +0.02, so little of the loss is released (16% of the gap for TabFM, none for the others). TabM, which has no column-stage labels, loses even more capture there. We therefore do not attribute the summary-induced harms to the pathway.
  • SGEMM. As you note, residual column labels make predictions worse: beyond-pair MSE rises in every arm, by 17.5% for TabPFN-3 and 27.2% for TabICLv2 with aligned inputs. We therefore do not attribute the runtime harms to the pathway either.
  • Weather, TabICLv2. Labels built from the context rows alone raise capture from 0.62 to 0.87 with 4,000 rows and from 0.85 to 0.94 with 16,000, and lower RMSE by 3.7% and 1.8%; we added them after registration. The registered labels subtract station-month means. Those means use every row of the station block except the draw's test rows, so they draw on labels outside the context; they raise capture to 0.77 and 0.91. A registered linear version beats its matched direction but not intact labels. With the covariates permuted, none of these swaps raises capture (post hoc).
  • Weather, TabPFN-3. Context-only labels raise capture from 0.48 to 0.74 with 1,000 rows and from 0.81 to 0.86 with 2,000, but RMSE does not improve: it changes by −0.9% [−4.6, 2.9] and +2.7%.
  • Food delivery, TabICLv2 (registered, new draws). Removing the age-and-rating fit from the column-stage labels releases about half of the loss (+0.34 [0.27, 0.41]) and lowers RMSE by 2.0% (descriptive).

The matched random directions are not neutral on the weather table, so Table 31 also reports each gain over intact labels.

Revised manuscript: Sections 5 and 7; Appendices H, I, P and Q; Tables 12, 13 and 27–33.

We hope these revisions address your concerns, and we would be very glad to clarify any remaining point.

Thank you for the detailed response and the new experiments. The weather table, the TabArena components and the Bayesian comparison mostly answer my questions, and the paper is now clear about what its results do not show. However, my main concern was whether crowding matters in practice, and the new results do not change that. On ordinary tables, crowding is rarely found and costs at most about 1% of test MSE. I will hold my rating.

Reviewer_xTETOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a phenomenon they call "crowding out" where informative features will crowd out features that form interactions which are known to contribute to the output. They establish an experimental procedure to test this using semi-synthetic data which uses a planted order-3 interaction on top of real tabular data in order to estimate this effect. They then design some metrics and run some experiments, afterwards claiming this "crowding out" effect occurs only on tabular foundation models.

Critical strengths

Critical Strengths

The problem of feature crowding is interesting, and underexplored on tabular foundation models.

According to Figure 1, the authors seem to discover some phenomenon which could be specific to TFMs.

Critical weaknesses

Critical Weaknesses

  1. The κ\kappa value, which they call the "capture", is the entire centerpiece of all experiments run and all conclusions drawn. This coefficient is given a few sentences as a definition and no further discussion on its choice is made. In particular, the noise assumptions and fitting with the c()c() function versus the yrealy_real value are not discussed. These choices are of critical importance because every table, figure, discussion, and conclusion is based on this choice. For instance, an alternative is to only fit the (1,g) terms because there is no guarantee to understand the underlying c()c() function which the model ultimately learns on real-world data.

  2. Next, the paired difference between the "capture" for permuted columns and original columns is defined as the "crowding". Nowhere is it described what this is intended to capture or the intuition behind why it is defined in this way.

  3. The bias towards lower-order interactions is known to occur in random forests and neural networks, so how this paper only discovers crowding in TFMs is confusing, and perhaps points further to the confusion about what is being uncovered by their paired difference metric which they call "crowding".

  4. The work seems disconnected from the existing literature. For example, the work uses "clues" to describe the lower-order components which give 'clues' for the higher-order components using the same features. This has historically been called the heredity of the interactions.

  5. Overall, given the lack of attention given to the intuition and meaning of the defined metrics, it becomes increasingly difficult to follow the conclusions made later in the paper, especially column label swapping and measured track. Although a potentially interesting phenomenon was discovered, it needs much more attention in understanding how to interpret the experimental results.

Questions for authors

See the weaknesses above. If necessary, please ask your friend to convert the above weaknesses into a list of corresponding questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xTET (1/2)

We sincerely thank the reviewer for pressing on the definitions that every result depends on. We agree that the submission's few sentences on them were hard to follow: Section 3.1 is rewritten, and Sections 4 and 5 now open with what each experiment changes. Shared new evidence is in the general response. We addressed each weakness as follows.

1. What capture measures, and whether the regressors matter

“For instance, an alternative is to only fit the (1,g) terms.”

Your alternative gives the same results: with (1,g)(1,g) alone, no Table 1 mean changes by more than 0.0055.

Capture κ\kappa is the fraction of the planted jump that a predictor reproduces. For g=±0.6g=\pm0.6, it is the difference in mean prediction between the two values of gg, divided by 1.2. A predictor that ignores the planted columns scores 0; the Bayes predictor scores 1. Capture describes the predictor, not the data, so it needs no noise assumption.

In the paper we regress predictions on (1,g,yreal)(1, g, y_{\rm real}), where yrealy_{\rm real} is the observed real target, not a fitted function cc. The planted columns are drawn independently of the real table, so the coefficient on gg has the same population value with or without yrealy_{\rm real}; including it only improves precision. We recomputed all 3,132 saved predictions both ways (they are in the supplement). Interval endpoints move by at most 0.010. One interval, for a cell that barely learns the parity in either arm, now includes zero.

On the new weather table, with nothing planted, regressing on the interaction alone changes none of the covariate contrasts by more than 0.005 (post hoc). For the estimated TabArena components the two regressions can disagree: two Holm-positive contrasts change sign, so there, by a rule adopted post hoc, a contrast counts only if both regressions give the same sign.

Revised manuscript: Section 3.1; Appendices A, P and Q.

2. What crowding is meant to capture

“Nowhere is it described what this is intended to capture or the intuition behind why it is defined in this way.”

It is the effect on capture of making the other columns informative, with everything else fixed.

The permuted and aligned arms share the target, the planted columns, the width and every column's marginal distribution. Permuting the real columns removes their information about the target. So crowding, κperm−κaligned\kappa_{\rm perm}-\kappa_{\rm aligned}, isolates that information, and dilution, κnone−κperm\kappa_{\rm none}-\kappa_{\rm perm}, isolates the effect of extra uninformative columns.

Informative columns explain part of the target and lower the noise around gg, so a learner that fits all inputs jointly should capture at least as much with them. Boosting and gradient-trained networks generally do: across 88 learnable cells their crowding averages −0.14. A positive value means that the information itself displaces the interaction. In Figure 1, TabPFN-3 captures 0.96, 0.76 and 0.35 in the three arms (dilution 0.20, crowding 0.41), and CatBoost 0.89, 0.74 and 0.84 (crowding −0.10).

Revised manuscript: Section 3.1 and its table of the three arms.

AuthorsOpenReview ↗

Response to Reviewer xTET (2/2)

3. Whether crowding is specific to TFMs, and the low-order bias

“The bias towards lower-order interactions is known to occur in random forests and neural networks, so how this paper only discovers crowding in TFMs is confusing.”

We agree that it is not specific to TFMs. Random forests and xRFM crowd as well (+0.21 and +0.20 with 16,000 rows; Table 1). For forests this is the known masking of pure interactions by marginal effects (Wright et al., 2016); what is new is that pretrained in-context learners show it too. The abstract now names them.

It is also distinct from the low-order bias, which makes an interaction hard to learn in every arm. With 16,000 rows an MLP captures 0.93 from the planted columns alone and 0.65 once uninformative columns are added, but 0.83 when those columns are informative; a random forest captures 0.96, 0.41 and 0.20. Low-order bias predicts the first drop, not the second. Post hoc, we made this a computation: an exact Bayesian learner whose prior makes the product term improbable never crowds. Where the penalty binds only in the permuted arm, informative columns raise its capture by 0.50–0.89, because they lower the residual variance and so raise the product's Bayes factor. Two interventions tie crowding to capacity allocation in forests and xRFM: swapping the labels that grow forest splits releases and induces it, and turning off xRFM's feature weighting removes it.

Revised manuscript: Abstract; Section 3.2; Appendices A, F and G.1; Table 10.

4. Heredity

“This has historically been called the heredity of the interactions.”

We agree and adopt the term. Section 2 has a new paragraph on heredity (Nelder, 1977; Hamada & Wu, 1992). Section 3.3 now describes our additive and pairwise “clues” as lower-order components that make a fixed interaction hereditary, and its heading calls them parent components; the text and figures keep “clue” as a short name. A pure parity has none; adding a 3% one raises TabICLv2's permuted capture from 0.42 to 0.84 but leaves a paired loss of 0.46.

Revised manuscript: Sections 2 and 3.3.

5. Label swaps and the measured track

“Overall, given the lack of attention given to the intuition and meaning of the defined metrics, it becomes increasingly difficult to follow the conclusions made later in the paper, especially column label swapping and measured track.”

A TFM reads the context labels twice: when it builds column representations, and when it predicts. A swap changes only the first copy; inputs and prediction labels stay identical. Removing the competitor from that copy raises TabICLv2's capture from 0.08 to 0.65, adding it lowers capture from 0.71 to 0.08, and controls of equal variance change it by at most 0.05 (Table 3).

The measured track asks the question of Section 3 without planting: the competitor is a summary feature that an analyst might append, compared with a permuted copy. On weather observations, with nothing planted, informative covariates lower TabICLv2's capture of an interaction of UTC hour and longitude, and a label swap raises it again; Table 31 lists every registered test, including the two that failed.

Revised manuscript: Sections 4 and 5; Appendix P; Table 31.

We hope these revisions address your concerns, and we would be very glad to clarify any remaining point.

Thank you for the clarification that regression against (1,g)(1,g) and (1,g,y)(1,g,y) results in similar conclusions. The important missing piece, however, is a description of what "capture" is trying to capture and why it is defined in the chosen manner. For instance, the magic number 3 is not explained, although explored in the appendix.

Thank you also for the table in Section 3.1 which helps to clarify your three experimental setups. I have a question about what is meant by "none; width and marginals as ALIGNED" in this table. Is it implying this setup has no information about yy?

Overall, I still feel the lack of paper's connections to the existing literature will make it hard for a reader to understand what ideas this paper is contributing over existing concepts (low-order bias, heredity, etc.)

AuthorsOpenReview ↗

Response to Reviewer xTET

We thank the reviewer for reading the revised Section 3.1 closely. We answer your points in order, and we have revised the PDF where the text was still unclear.

1. What capture is for, and why it is a coefficient

“a description of what "capture" is trying to capture and why it is defined in the chosen manner”

Section 3.1 defined capture without saying what it is for; it now opens with this. Capture answers a question that test error cannot: how much of a known part of the target do the predictions reproduce? When other columns become informative, RMSE falls whether or not the interaction is still used. On the weather table the covariates cut TabICLv2's RMSE by about a quarter while lowering its capture by 0.07–0.17.

We therefore use the slope of held-out predictions on gg. It is 0 for a predictor that ignores the interaction's columns and 1 for one that reproduces gg in full. Because gg is independent of the other columns, adding any function of them to a predictor leaves the slope unchanged, whereas RMSE or a correlation changes between arms for any learner that uses the real columns. The slope changes only if the use of the interaction changes.

2. Why three inputs

“the magic number 3 is not explained, although explored in the appendix”

You are right that the paper gave no reason; Section 3.1 now does. Order three is the lowest at which a parity has neither main effects nor pairwise structure (Appendix M.1). It is a choice of the benchmark, not part of the definition, and the findings do not depend on it: TabICLv2 and TabFM also crowd a pairwise parity (0.76 → 0.56 and 0.99 → 0.53; Section 3.3), and both unplanted cases are pairwise (G1, G2).

3. The PERM row of the table

“what is meant by "none; width and marginals as ALIGNED" in this table”

About yrealy_{\rm real}, yes; about the whole target, no. The target is y=g+yrealy=g+y_{\rm real}. In PERM each real column is shuffled across rows, so the inputs say nothing about yrealy_{\rm real}, which stays in yy and acts as noise. The planted columns are never shuffled, so the inputs still determine gg in all three arms. The capture regressors are likewise (1,g,yreal)(1, g, y_{\rm real}).

The rest of that cell said what PERM keeps: as many columns as ALIGNED, with the same values in each. The cell was unclear because it made two statements at once; its column now reads Inputs inform about: gg only (NONE), gg only (PERM), gg and yrealy_{\rm real} (ALIGNED).

4. What the paper adds to existing concepts

“what ideas this paper is contributing over existing concepts (low-order bias, heredity, etc.)”

We agree that the paper left this comparison to the reader; the new Table 34 makes it. In short, the existing concepts explain why an interaction is hard to learn, whereas crowding is a learner losing an interaction that it does learn, because other columns became useful.

  • Low-order bias explains low capture in NONE and PERM, not a loss when the same columns become informative: an MLP gains (0.65 → 0.83), while TabPFN-3 falls from 0.76 to 0.35.
  • Irrelevant columns hurt, which is dilution; our result concerns relevant columns at the same width.
  • Heredity describes targets, not what a learner loses. Parents make the parity easier to learn, yet TabICLv2 still loses 0.46.
  • Simplicity bias and shortcuts mostly concern gradient training on redundant features. Here no weight is trained and the interaction is complementary.
  • Masking in forests (Wright et al., 2016) is the closest precedent, and we reproduce it.

5. Why we think it matters

  • It is new. Three independently developed TFMs, with frozen weights, lose a learnable interaction when other columns become informative, while boosting and most gradient-trained networks gain from the same columns.
  • It has a causal handle. Changing only the labels that build column representations turns the loss on and off (TabICLv2: 0.08 → 0.65 and 0.71 → 0.08; controls of equal variance: at most 0.05).
  • It occurs without planting. On weather observations TabICLv2 shows the same loss in two held-out samples of stations. A label swap releases it and, with context-only labels, lowers RMSE by 3.7% (G1).
  • Accuracy hides it. There RMSE improves by about a quarter while capture falls, so a benchmark that ranks by error cannot show it.

Your review's first concern was the choice of regressors, and we are grateful that you looked at our answer. We hope these replies resolve your remaining points.

Revised manuscript: Sections 2 and 3.1; Table 34 (Appendix R).

We would be very glad to clarify anything that is still unclear.

Reviewer_t1zCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a crowding out phenomenon in tabular foundation models (TFMs), whereby informative features suppress the learning of higher-order interactions. The authors distinguish this phenomenon from feature dilution through controlled experiments and use label-swapping interventions to investigate its mechanism. They further introduce CrowdBench and propose a residual-based method to recover suppressed interactions.

Critical strengths
  1. This paper identifies an interesting failure mode of TFMs, offering new insights into feature competition in in-context learning.

  2. The controlled experimental design effectively distinguishes crowding out from dilution while examining different interaction orders, feature widths, and context sizes.

  3. Label-swapping experiments and representation-level interventions provide valuable causal evidence for the role of in-context labels in interaction suppression.

  4. The evaluation covers multiple TFMs and traditional baselines, and CrowdBench and the proposed recovery method enhance the practical value of this work.

Critical weaknesses
  1. The main findings rely heavily on manually implanted parity interactions, and their generalizability to naturally occurring interactions has not been sufficiently established.

  2. Although the label-swapping experiments reveal a causal pathway, the underlying computational mechanisms remain incompletely explained, especially across different TFM architectures.

  3. When overall predictive performance improves, the practical significance of reduced interaction capture is unclear, as observed in some real-world experiments.

  4. The proposed recovery method relies on a predefined interaction dictionary, and its improvements on standard datasets largely overlap with generic residual stacking.

Questions for authors
  1. In real-world datasets without manually implanted interactions, how frequently does crowding out occur?

  2. Can the authors further disentangle the contributions of attention routing and feature representations across different TFM architectures?

  3. When interaction capture decreases but RMSE improves, what practical consequences do the suppressed interactions have?

  4. Compared with simpler residual correction methods, how does the proposed recovery method perform in terms of effectiveness and computational efficiency?

3 comments
AuthorsOpenReview ↗

Response to Reviewer t1zC

We sincerely thank the reviewer for the constructive feedback and for recognizing the matched design and the causal evidence from label swaps; your questions led us to test crowding where nothing is planted. The shared new evidence is in the general response (G1–G5). We addressed each question as follows.

1. Natural interactions

“In real-world datasets without manually implanted interactions, how frequently does crowding out occur?”

We cannot give a general frequency. On ordinary tables crowding is rarely detected and small; without planting, it does occur on a weather table.

  • Unplanted (G1–G3). On TabArena the TFMs mostly capture a stable pairwise component; 3 of 45 contrasts stay positive after Holm adjustment and a post hoc rule that the sign must hold when predictions are regressed on the component alone. On the weather table, informative covariates lower TabICLv2's capture of local time by 0.07–0.17, and a column-stage label swap releases the loss; 11 of 13 registered tests pass (Table 31). Among 13 components that a calibrated test shows the TFMs leave unused, the all-columns contrast detects no crowding (4 are dilution), a null that says little about absence.
  • Planted, beyond parity (exploratory pilot; Table 25). TabICLv2 and TabFM crowd cyclic sums (0.30–0.49) and matches of categorical columns (0.46 and 0.66); TabPFN-3 crowds two of six shapes.

Revised manuscript: Section 5; Appendices L, P and Q.

2. Routing and representations across architectures

“Can the authors further disentangle the contributions of attention routing and feature representations across different TFM architectures?”

We added no experiment here: transplant points differ by architecture, and we prioritized the natural-interaction tests of your first question. The decomposition is incomplete, and Section 4 now states its scope.

  • All three TFMs. Swapping the labels seen by the column or cell encoder releases and induces crowding; controls of equal variance do not (Table 3).
  • TabPFN-3. The swap releases 0.416 in capture. Transplanting the donor's values and query skip alone restores 0.094, adding its attention routing restores the remaining 0.322, and routing alone restores 0.037 (Table 5).
  • TabICLv2. Transplanting all column-group representations from a swapped donor transfers the release (0.045 → 0.668); the groups that contain the support do not do so alone (0.040).
  • TabFM. We have the swaps only.

Revised manuscript: Section 4; Appendices D and E.

3. Consequences when capture falls but RMSE improves

“When interaction capture decreases but RMSE improves, what practical consequences do the suppressed interactions have?”

Section 7 now names three cases (“When crowding matters”):

  • Unused accuracy. On the weather table, giving TabICLv2 local time as a column lowers its RMSE by 5–11% where it crowds; this bounds all its under-use, not crowding alone. On TabArena, crowded unplanted components cost at most 1.3% of test MSE.
  • The interaction as the estimand. On CH65 the fitted third-order coefficient shrinks by 0.15–0.18 of its measured value while RMSE improves by 12–15%, so reading the interaction off the fitted model would under-report it.
  • Reversal with context size. An RMSE gain at one context size need not hold at another: on SGEMM and KTT, summaries improve every learner's RMSE with 128 rows but can raise it with longer contexts (TFM RMSE +24–31% in one KTT cell). TabM shares that harm, so this shows the reversal, not its cause.

Revised manuscript: Section 7.

4. Recovery versus simpler residual correction

“Compared with simpler residual correction methods, how does the proposed recovery method perform in terms of effectiveness and computational efficiency?”

On planted interactions it beats generic stacking at negligible cost; on native tables it does not, and we claim no recovery there.

  • Effectiveness. Generic stacking refits the same TFM to cross-fitted residuals on all columns. Averaged by model and context size over the aligned parity fits, it lowers RMSE by 0.1–11.8%; refitting on the discovered columns lowers it by 21.9–40.6%, within 1.3 points of the oracle support. In separate-sample controls, ridge on the single discovered product matches the TFM refit, so finding the support, not the refit learner, drives recovery.
  • Efficiency. The search takes a median 0.03 s (4,000 rows) or 0.08 s (16,000), against 7–17 s per TFM fit.
  • What remains. A diagnostic that locates an omitted interaction within a stated dictionary (products of three columns), which stacking cannot do.

Revised manuscript: Section 6; Appendices K and K.1; Table 18.

We hope these revisions address your concerns, and we would be very glad to clarify any remaining point.

Thank you for the detailed response and additional experiments. The revisions address several of my concerns. I appreciate the authors' efforts and will maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer t1zC

We thank the reviewer for reading the response and the new experiments. Your first question led to the tests on unplanted interactions, which changed what the paper claims. If any of your concerns remains open, we would be glad to address it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.