- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates the question whether we can avoid any computational overhead in tabular foundation models by skipping layer computations. For that, the authors suggest to use a affine transformation from a starting layer to and end layer showing both, a substantial amount of a stacked architecture can be replaced while retaining the predictive power of the full architecture.
In their empirical evaluation, they demonstrate the reduction across various model families and a thorough selection of 171 classification datsets. Their evaluation shows minimal predictive degradation answering the question open that it works, but leaving room for the answer of the why.
Critical strengths
S1: memory and computational footprint analysis
The authors provide in Table 1 an analysis on the hardware impact of their proposed approach of skipping layers via an affine transformation. It shows that a substantial amount of parameter and computational overhead can be saved. However, in this table, I’m missing the performance gap to the full architecture to get the full picture.
S2: Thorough analysis of the core claim of skipping layers via an affine transformation
The paper demonstrates the effect of skipping layers via the proposed affine transformation without loosing too much of the predictive performance. The analysis in Section 7 provides a natural step and answer the follow-up question right away, where apparently the community is prone towards using more model capacity than actually needed to reach the same level of predictive performance evaluated on the datasets that are used in the empirical study.
Critical weaknesses
W1: Empirical study does not justify the ‘why’ it works.
One of my major concerns with the paper is, that even though the authors provide good indications that leveraging the affine transformation seems to work, there is a lacking behind the answer ‘why’ it actually works. In Section 4, the authors provide an discussion of the similarities across the block representations extracted from sota tfms. The section ands with “representation similarity therefore does not accurately predict replacedibility”. Leveraging the affine transformation from a starting layer (s) to an end layer (e) , then what is it what actually steers the predictive power? I’d be curious to see a discussion on the potential effects and reasons the authors figured out during their evaluation. It appears a bit of an interesting insight where we are lacking of the reasons.
W2: no analysis on the characteristics where skipping layers work.
The though of W1 directly leads me to a second weakness, where we are try to solve the answer by inspecting for exampling any data characteristics showing effects where we indeed cannot skip too much of the layers and potentially need a full stack to cipher the concrete effects of feature-target relationships. Do we see dataset characteristics where the central claim of TFM are shallow architectures under the hood does not longer hold. Can we save more computational power just for datasets, where the relationships among the columns of a table are more or less pretty straightforward. How the divide the datasets complexities in a meaningful manner such that we see such effects, where there are datasets that are harder to encode. Such an more in-depth analysis in not presented and we are rather left with the question of W1 - why it works. So what does an meta feature analysis from the datasets potentially provide as information, are we dealing here in most cases with a low intrinisic dimensionality where we indeed can encode it with less amount of parameters, what about the number of informative features etc.
W3: OOD robustness
I’m also a bit curious about the ood robustness, where the test distributions can also not to be expected to align with the iid training samples. Therefore, the intersting part starts at B.14 in the appendix, where we then see “transfer weakens when the schema differ, even within one domain”. However, we are also now having the question in the room what happens with the generalization capabilities with some distribution shifts. Why does it matter? The authors propose an affine transaltor that is fitted on samples form a specific calibration distribution. Taking this into account and thinking one step further, what happened under changes of feature distribution, feature-target relationships, or any other changes in the feature distribution like we have it for any tabular data storing time-dependent data. If in shallow representation the ‘obvious’ feature-target relationships are encoded, do we still need deeper models to encode distribution shifts?
Minor:
- at the end of related works, there is “.” missing
Questions for authors
- Q1: Can the authors provide a discussion on the ‘why’ (W1)?
- Q2: Did the authors inspect in the conducted experiment any further analysis on the meta-features, feature-target relationships, feature distribution shifts (W2)?
- Q3: Can we have a more thorough analysis on the ood robustness (W3)?
4 comments
2/2
... what happened under changes of feature distribution, feature-target relationships, or any other changes ... for any tabular data storing time-dependent data. (W3, Q3)
We now test the three shifts the reviewer names in a new appendix section (App. B.12, Table 19), on the eight clinical datasets (the temporal one on MIMIC-III), for two dual-axis and two single-axis models and three seeds. In every case, the full model faces the same shift, and we compare the gap between the compressed and the full model with and without shift. The map is either reused unchanged from the source data or refitted without labels on the shifted adaptation rows (the calibration rows of the submission), which are separate from the test rows.
First, under a shift of the feature distribution (the three most informative features moved by up to two standard deviations), the full models themselves lose 0.04–0.08 AUROC, and the single-axis models lose the same at half depth. The dual-axis models lose about 0.03 more on average, but this is driven by a few datasets (median gap of at most 0.004). Second, under a change of the feature–target relation (labels redrawn with the effect of the strongest feature reversed), a map reused from the source barely affects the single-axis models at half depth (−0.003 and −0.008) but fails for the dual-axis ones (TabPFNv2: −0.113). Refitting it without labels, which takes seconds, brings every model and depth back to, or within 0.01 of, its gap without shift. Third, the temporal shift, from an earlier period (2001–2008) to a later one (2008–2012) of MIMIC-III, is mild: the full models lose 0.004–0.022, and the gaps move by at most 0.009. A second test on MIMIC-IV, where the recording system does not change (admissions of 2008–2012 against 2017–2019), gives the same picture.
... do we still need deeper models to encode distribution shifts? (W3)
Not at half depth, for the shifts we tested: the compressed models largely track the full ones under the feature and temporal shifts with the map reused unchanged, and under a changed feature–target relation once the map is refitted on unlabeled rows. With one block kept, the gap of a map reused from the source also grows by at most 0.009 under the feature and temporal shifts.
... in this table, I'm missing the performance gap to the full architecture to get the full picture. (S1)
We thank the reviewer for this suggestion. Table 1 now reports the ΔAUROC on MIMIC-III next to the hardware numbers, so that the savings and the accuracy cost can be read together: at half depth, every model stays within 0.008 AUROC of its full version.
at the end of related works, there is "." missing (Minor)
We thank the reviewer for spotting this, and we fixed it.
We hope the additional analysis have addressed all of the reviewer's concerns and remain available for further clarifications.
[1] Facco, d'Errico, Rodriguez, Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 2017.
First of all, thank you very much for providing a response to my raised concerns and questions regarding the proposed work. Let us have a closer look at them together:
Q1.
Thanks for providing a more thorough discussion. But the new insights also raises more questions about the proposed method. If “the information the prediction needs is already present before the replaced window”, and a logistic readout is sufficient, then it seems like there is an even more lightweighted approach to solve the problem at hand. What is then the practical advantage of the proposed affine translator compared to directly approximating the output. Second, also if the testbeds just encompasses settings with a rather low intrinsic dimensionality, then I’d be curious to see settings where the intrinsic dimensionality approximates the feature dimensionality at hand and if the proposed approach still is applicable and the claim that TFM are effectively shallow still holds. From my humble perspective, this point is not fully resolved and we need to spend more time to understand these dynamics.
Q2.
Thanks for providing the new insights. The AUROC gap between a tuned GBDT and a logistic regression serves as a rough proxy; i would not assume it to be a “consistent predictor” - but I’m open for references justifying it. Do we have a reference where this is indeed a measure for feature-target (non-)linearities?
Q3.
Ok, that needs a closer inspection. So with the shift of the feature distribution, apparently it is the case that the performance is rather model-specific than a claim that holds in general. For cases where the affine map does not transfer, only a refitting recovers the performance. Therefore, for any deployment settings one might need a fallback to a re-calibration. Also, I’d be curious to inspect settings where not only the magnitude of informative features change but what happens with completely new feature-target interactions where some spurious features might get more important and vice versa.
Minor: Acknowledged.
Thank you for taking time to answer all the questions.
We thank the reviewer for the careful reading of our response and for the follow-up questions.
What is then the practical advantage of the proposed affine translator compared to directly approximating the output. (Q1)
For classification on a single dataset, a readout that directly approximates the output is as accurate as our map (App. B.13). At prediction time, the readout only skips the map and the head, since both keep the same blocks. We see this result as further support for the main finding of the paper, that most of the depth is not needed on a given task. The map has three practical advantages. First, it needs no new output layer: the pre-trained head is kept, so the same map also applies to regressors (Sec. 5.2). Second, for the single-axis models, one map can be pooled over datasets and applied to a new dataset without any fit or full-model pass (Table 26), whereas a readout is tied to the classes of one dataset. Third, it can replace any window of blocks and not only the last ones, which allows us to study where each model uses its depth (Sec. 4).
... settings where the intrinsic dimensionality approximates the feature dimensionality at hand ... (Q1)
Our 171 datasets already include such settings, and the approach remains applicable on them. In the analysis of App. B.3, the median dataset needs 75% as many principal components as it has features to explain 95% of its variance, and 30 datasets need at least 90%. A new analysis restricted to these 30 datasets shows that half depth costs at most 0.004 mean AUROC for every model, the same as on the remaining datasets (at most 0.005). We also clarify that what is low-dimensional is the update of the replaced blocks, not the data. We agree with the reviewer that why pre-training produces this structure is not yet understood (as stated in App. B.3) and left for future work.
... serves as a rough proxy; i would not assume it to be a "consistent predictor" - but I'm open for references justifying it. (Q2)
We agree that this gap is a rough proxy, and now refer to it as such in the paper. With "consistent" we only meant that its correlation with the cost has the same sign and is significant for all eight models. Silvey and Liu [1] use the same kind of gap: they define the non-linearity of a dataset as the AUC difference between XGBoost and a logistic regression. It is also related to measuring the linear separability of a classification problem by the error of a linear classifier [2]. Our analysis also includes a second proxy that involves no linear model, the depth of the tuned GBDT. Datasets that need deeper trees also lose more, significantly for seven of the eight models at half depth and with one block kept. We added this result in the revised pdf (App. B.3).
... the performance is rather model-specific than a claim that holds in general. (Q3)
At half depth and on six of the eight datasets, all four models lose at most 0.02 AUROC relative to their full version at every magnitude of the shift. The larger average gap of the dual-axis models comes from the remaining two datasets (SUPPORT2 and Thyroid). We agree that these models are less robust there, which is why Table 19 reports each model separately.
... for any deployment settings one might need a fallback to a re-calibration. (Q3)
Yes, when the feature–target relation changes, the map should be refitted, in particular for the dual-axis models. But this needs no additional labels and takes seconds (Table 32). In this case, the full model also needs a new support set to learn the new relation in context and the refit only adds one forward pass and a closed-form fit on this new context.
... completely new feature-target interactions where some spurious features might get more important and vice versa. (Q3)
Following the reviewer's suggestion, we ran two new experiments with the same setting of Table 19, and in both of them a refit with no additional labels is again sufficient. In the first, the feature with the weakest effect on the source becomes the most important one and the strongest one loses its effect. In the second, an interaction between the two strongest features, absent on the source, is added to the relation. At half depth, a map reused from the source costs the single-axis models at most 0.006 AUROC and the dual-axis ones up to 0.040 (TabPFNv2, when the two features exchange their roles). After the refit, every model is within 0.005 of its gap without shift, also with one block kept. We added this result in the revised pdf (App. B.12).
We thank the reviewer again for the constructive discussion that helped improving the paper and remain available for further clarifications.
[1] Silvey, Liu. Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study. Journal of Medical Internet Research, 2024.
[2] Ho, Basu. Complexity Measures of Supervised Classification Problems. IEEE TPAMI, 2002.
1/2
We thank the reviewer for the valuable feedback, and for recognizing that "a substantial amount of parameter and computational overhead can be saved". We are also grateful that the reviewer appreciated the "thorough analysis of the core claim of skipping layers via an affine transformation". We address each point below, where section and table numbers refer to the revised PDF.
We thank the reviewer for this question, and we agree that the 'why' deserves a discussion. We added a new analysis (App. B.3, Table 7, and summarized in Sec. 4) that points to three observations. First, the information the prediction needs is already present before the replaced window: in a new experiment (App. B.13, Table 21), a logistic readout of the window's input, fitted to the full model's predicted probabilities, reproduces them about as well as our map at every depth. Second, the update applied by the replaced blocks is large but low-dimensional: it is one to six times as large as its input, consistent with the failure of simply dropping the blocks, but it is concentrated in a few directions (effectively 2–27 of the 192–2,048 dimensions of the hidden state). Third, within a single dataset, this update is close to affine: on unseen hidden states, an affine map of the window's input explains a large share of its variance (a median of 65–98% at half depth and 42–95% with one block kept). As a consistency check, the datasets with a more affine update also lose less with one block kept (Spearman ρ = 0.49).
These results suggest that the predictive power is steered by the representation built before the window, and that the map reproduces the few directions along which the replaced blocks move it. This is also consistent with the cosine similarity of Sec. 4 being a poor predictor of replaceability: it is not invariant to affine maps, so two representations can have a low cosine similarity and still be related by an affine map. Why pre-training produces this structure remains an interesting question for future work.
Following the reviewer's suggestion, we added a meta-feature analysis over the 171 datasets (App. B.3, Table 7; summarized in Sec. 4). It relates the cost of compression to each characteristic through partial Spearman correlations that account for the numbers of rows, features and classes. The most consistent predictor of the cost is the non-linearity of the feature–target relation, which we measure as the AUROC gap between a tuned GBDT and a logistic regression; this gap also gives a simple way to divide the datasets by complexity. More non-linear datasets lose more for every model, and the correlation with ΔAUROC is stronger with one block kept (ρ from −0.36 to −0.55) than at half depth (from −0.24 to −0.41). At half depth, however, the claim holds on every group of datasets: the cost of every model stays within 0.011 in each third of the datasets, from the most to the least linear. With one block kept, the least linear datasets do need more of the stack: for example, the cost of TabFM grows from −0.006 on the most linear third to −0.059 on the least linear third.
In the new meta-feature analysis (App. B.3), intrinsic dimension, the number of features, the number and share of informative features, and the difficulty of the dataset do not predict the cost consistently. We estimated the intrinsic dimension, which reflects how redundant the columns are, with TwoNN [1] and PCA. After correction for multiple testing, it is significantly correlated with the cost only for LimiX-16M with one block kept, and there the datasets with a higher intrinsic dimension lose less, not more. A low intrinsic dimension of the data is therefore not what makes compression work. What is low-dimensional is instead the update of the replaced blocks (see our answer to Q1).
Yes, and the label-free KL tolerance (Eq. 3, App. B.5) already does so automatically: a new analysis (App. B.3) shows that, for most models, it removes more of the stack where the relation is simpler, i.e., with fewer classes, shallower tuned GBDTs and a smaller gap between the GBDT and the logistic regression. For TabPFN-3, TabICL and TabFM, the share of the stack it removes differs by 21–27 percentage points between the third of the datasets with the shallowest tuned GBDTs and the third with the deepest.