acceptodds
Under review as a conference paper at ICLR 2027

Tabular Foundation Models Are Effectively Shallow

Abstract

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: *can TFMs be simplified by removing architectural redundancies without sacrificing performance?* To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to at a median cost of AUROC, while a single-block TabPFN-3 serves cached queries faster using less memory at negligible cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Learning, Tabular Foundation Models, Transformers, Healthcare

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_cu6BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the question whether we can avoid any computational overhead in tabular foundation models by skipping layer computations. For that, the authors suggest to use a affine transformation from a starting layer to and end layer showing both, a substantial amount of a stacked architecture can be replaced while retaining the predictive power of the full architecture.

In their empirical evaluation, they demonstrate the reduction across various model families and a thorough selection of 171 classification datsets. Their evaluation shows minimal predictive degradation answering the question open that it works, but leaving room for the answer of the why.

Critical strengths

S1: memory and computational footprint analysis

The authors provide in Table 1 an analysis on the hardware impact of their proposed approach of skipping layers via an affine transformation. It shows that a substantial amount of parameter and computational overhead can be saved. However, in this table, I’m missing the performance gap to the full architecture to get the full picture.

S2: Thorough analysis of the core claim of skipping layers via an affine transformation

The paper demonstrates the effect of skipping layers via the proposed affine transformation without loosing too much of the predictive performance. The analysis in Section 7 provides a natural step and answer the follow-up question right away, where apparently the community is prone towards using more model capacity than actually needed to reach the same level of predictive performance evaluated on the datasets that are used in the empirical study.

Critical weaknesses

W1: Empirical study does not justify the ‘why’ it works.

One of my major concerns with the paper is, that even though the authors provide good indications that leveraging the affine transformation seems to work, there is a lacking behind the answer ‘why’ it actually works. In Section 4, the authors provide an discussion of the similarities across the block representations extracted from sota tfms. The section ands with “representation similarity therefore does not accurately predict replacedibility”. Leveraging the affine transformation from a starting layer (s) to an end layer (e) , then what is it what actually steers the predictive power? I’d be curious to see a discussion on the potential effects and reasons the authors figured out during their evaluation. It appears a bit of an interesting insight where we are lacking of the reasons.

W2: no analysis on the characteristics where skipping layers work.

The though of W1 directly leads me to a second weakness, where we are try to solve the answer by inspecting for exampling any data characteristics showing effects where we indeed cannot skip too much of the layers and potentially need a full stack to cipher the concrete effects of feature-target relationships. Do we see dataset characteristics where the central claim of TFM are shallow architectures under the hood does not longer hold. Can we save more computational power just for datasets, where the relationships among the columns of a table are more or less pretty straightforward. How the divide the datasets complexities in a meaningful manner such that we see such effects, where there are datasets that are harder to encode. Such an more in-depth analysis in not presented and we are rather left with the question of W1 - why it works. So what does an meta feature analysis from the datasets potentially provide as information, are we dealing here in most cases with a low intrinisic dimensionality where we indeed can encode it with less amount of parameters, what about the number of informative features etc.

W3: OOD robustness

I’m also a bit curious about the ood robustness, where the test distributions can also not to be expected to align with the iid training samples. Therefore, the intersting part starts at B.14 in the appendix, where we then see “transfer weakens when the schema differ, even within one domain”. However, we are also now having the question in the room what happens with the generalization capabilities with some distribution shifts. Why does it matter? The authors propose an affine transaltor that is fitted on samples form a specific calibration distribution. Taking this into account and thinking one step further, what happened under changes of feature distribution, feature-target relationships, or any other changes in the feature distribution like we have it for any tabular data storing time-dependent data. If in shallow representation the ‘obvious’ feature-target relationships are encoded, do we still need deeper models to encode distribution shifts?

Minor:

  • at the end of related works, there is “.” missing
Questions for authors
  • Q1: Can the authors provide a discussion on the ‘why’ (W1)?
  • Q2: Did the authors inspect in the conducted experiment any further analysis on the meta-features, feature-target relationships, feature distribution shifts (W2)?
  • Q3: Can we have a more thorough analysis on the ood robustness (W3)?
4 comments

We thank the reviewer for the valuable feedback, and for recognizing that "a substantial amount of parameter and computational overhead can be saved". We are also grateful that the reviewer appreciated the "thorough analysis of the core claim of skipping layers via an affine transformation". We address each point below, where section and table numbers refer to the revised PDF.

Can the authors provide a discussion on the 'why'? (W1, Q1)

We thank the reviewer for this question, and we agree that the 'why' deserves a discussion. We added a new analysis (App. B.3, Table 7, and summarized in Sec. 4) that points to three observations. First, the information the prediction needs is already present before the replaced window: in a new experiment (App. B.13, Table 21), a logistic readout of the window's input, fitted to the full model's predicted probabilities, reproduces them about as well as our map at every depth. Second, the update applied by the replaced blocks is large but low-dimensional: it is one to six times as large as its input, consistent with the failure of simply dropping the blocks, but it is concentrated in a few directions (effectively 2–27 of the 192–2,048 dimensions of the hidden state). Third, within a single dataset, this update is close to affine: on unseen hidden states, an affine map of the window's input explains a large share of its variance (a median of 65–98% at half depth and 42–95% with one block kept). As a consistency check, the datasets with a more affine update also lose less with one block kept (Spearman ρ = 0.49).

These results suggest that the predictive power is steered by the representation built before the window, and that the map reproduces the few directions along which the replaced blocks move it. This is also consistent with the cosine similarity of Sec. 4 being a poor predictor of replaceability: it is not invariant to affine maps, so two representations can have a low cosine similarity and still be related by an affine map. Why pre-training produces this structure remains an interesting question for future work.

Do we see dataset characteristics where the central claim ... does not longer hold. (W2, Q2)

Following the reviewer's suggestion, we added a meta-feature analysis over the 171 datasets (App. B.3, Table 7; summarized in Sec. 4). It relates the cost of compression to each characteristic through partial Spearman correlations that account for the numbers of rows, features and classes. The most consistent predictor of the cost is the non-linearity of the feature–target relation, which we measure as the AUROC gap between a tuned GBDT and a logistic regression; this gap also gives a simple way to divide the datasets by complexity. More non-linear datasets lose more for every model, and the correlation with ΔAUROC is stronger with one block kept (ρ from −0.36 to −0.55) than at half depth (from −0.24 to −0.41). At half depth, however, the claim holds on every group of datasets: the cost of every model stays within 0.011 in each third of the datasets, from the most to the least linear. With one block kept, the least linear datasets do need more of the stack: for example, the cost of TabFM grows from −0.006 on the most linear third to −0.059 on the least linear third.

... are we dealing here in most cases with a low intrinsic dimensionality ..., what about the number of informative features etc. (W2)

In the new meta-feature analysis (App. B.3), intrinsic dimension, the number of features, the number and share of informative features, and the difficulty of the dataset do not predict the cost consistently. We estimated the intrinsic dimension, which reflects how redundant the columns are, with TwoNN [1] and PCA. After correction for multiple testing, it is significantly correlated with the cost only for LimiX-16M with one block kept, and there the datasets with a higher intrinsic dimension lose less, not more. A low intrinsic dimension of the data is therefore not what makes compression work. What is low-dimensional is instead the update of the replaced blocks (see our answer to Q1).

Can we save more computational power just for datasets, where the relationships among the columns of a table are more or less pretty straightforward. (W2)

Yes, and the label-free KL tolerance (Eq. 3, App. B.5) already does so automatically: a new analysis (App. B.3) shows that, for most models, it removes more of the stack where the relation is simpler, i.e., with fewer classes, shallower tuned GBDTs and a smaller gap between the GBDT and the logistic regression. For TabPFN-3, TabICL and TabFM, the share of the stack it removes differs by 21–27 percentage points between the third of the datasets with the shallowest tuned GBDTs and the third with the deepest.

... what happened under changes of feature distribution, feature-target relationships, or any other changes ... for any tabular data storing time-dependent data. (W3, Q3)

We now test the three shifts the reviewer names in a new appendix section (App. B.12, Table 19), on the eight clinical datasets (the temporal one on MIMIC-III), for two dual-axis and two single-axis models and three seeds. In every case, the full model faces the same shift, and we compare the gap between the compressed and the full model with and without shift. The map is either reused unchanged from the source data or refitted without labels on the shifted adaptation rows (the calibration rows of the submission), which are separate from the test rows.

First, under a shift of the feature distribution (the three most informative features moved by up to two standard deviations), the full models themselves lose 0.04–0.08 AUROC, and the single-axis models lose the same at half depth. The dual-axis models lose about 0.03 more on average, but this is driven by a few datasets (median gap of at most 0.004). Second, under a change of the feature–target relation (labels redrawn with the effect of the strongest feature reversed), a map reused from the source barely affects the single-axis models at half depth (−0.003 and −0.008) but fails for the dual-axis ones (TabPFNv2: −0.113). Refitting it without labels, which takes seconds, brings every model and depth back to, or within 0.01 of, its gap without shift. Third, the temporal shift, from an earlier period (2001–2008) to a later one (2008–2012) of MIMIC-III, is mild: the full models lose 0.004–0.022, and the gaps move by at most 0.009. A second test on MIMIC-IV, where the recording system does not change (admissions of 2008–2012 against 2017–2019), gives the same picture.

... do we still need deeper models to encode distribution shifts? (W3)

Not at half depth, for the shifts we tested: the compressed models largely track the full ones under the feature and temporal shifts with the map reused unchanged, and under a changed feature–target relation once the map is refitted on unlabeled rows. With one block kept, the gap of a map reused from the source also grows by at most 0.009 under the feature and temporal shifts.

... in this table, I'm missing the performance gap to the full architecture to get the full picture. (S1)

We thank the reviewer for this suggestion. Table 1 now reports the ΔAUROC on MIMIC-III next to the hardware numbers, so that the savings and the accuracy cost can be read together: at half depth, every model stays within 0.008 AUROC of its full version.

at the end of related works, there is "." missing (Minor)

We thank the reviewer for spotting this, and we fixed it.

We hope the additional analysis have addressed all of the reviewer's concerns and remain available for further clarifications.


[1] Facco, d'Errico, Rodriguez, Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 2017.

First of all, thank you very much for providing a response to my raised concerns and questions regarding the proposed work. Let us have a closer look at them together:

Q1.

Thanks for providing a more thorough discussion. But the new insights also raises more questions about the proposed method. If “the information the prediction needs is already present before the replaced window”, and a logistic readout is sufficient, then it seems like there is an even more lightweighted approach to solve the problem at hand. What is then the practical advantage of the proposed affine translator compared to directly approximating the output. Second, also if the testbeds just encompasses settings with a rather low intrinsic dimensionality, then I’d be curious to see settings where the intrinsic dimensionality approximates the feature dimensionality at hand and if the proposed approach still is applicable and the claim that TFM are effectively shallow still holds. From my humble perspective, this point is not fully resolved and we need to spend more time to understand these dynamics.

Q2.

Thanks for providing the new insights. The AUROC gap between a tuned GBDT and a logistic regression serves as a rough proxy; i would not assume it to be a “consistent predictor” - but I’m open for references justifying it. Do we have a reference where this is indeed a measure for feature-target (non-)linearities?

Q3.

Ok, that needs a closer inspection. So with the shift of the feature distribution, apparently it is the case that the performance is rather model-specific than a claim that holds in general. For cases where the affine map does not transfer, only a refitting recovers the performance. Therefore, for any deployment settings one might need a fallback to a re-calibration. Also, I’d be curious to inspect settings where not only the magnitude of informative features change but what happens with completely new feature-target interactions where some spurious features might get more important and vice versa.

Minor: Acknowledged.


Thank you for taking time to answer all the questions.

We thank the reviewer for the careful reading of our response and for the follow-up questions.

What is then the practical advantage of the proposed affine translator compared to directly approximating the output. (Q1)

For classification on a single dataset, a readout that directly approximates the output is as accurate as our map (App. B.13). At prediction time, the readout only skips the map and the head, since both keep the same blocks. We see this result as further support for the main finding of the paper, that most of the depth is not needed on a given task. The map has three practical advantages. First, it needs no new output layer: the pre-trained head is kept, so the same map also applies to regressors (Sec. 5.2). Second, for the single-axis models, one map can be pooled over datasets and applied to a new dataset without any fit or full-model pass (Table 26), whereas a readout is tied to the classes of one dataset. Third, it can replace any window of blocks and not only the last ones, which allows us to study where each model uses its depth (Sec. 4).

... settings where the intrinsic dimensionality approximates the feature dimensionality at hand ... (Q1)

Our 171 datasets already include such settings, and the approach remains applicable on them. In the analysis of App. B.3, the median dataset needs 75% as many principal components as it has features to explain 95% of its variance, and 30 datasets need at least 90%. A new analysis restricted to these 30 datasets shows that half depth costs at most 0.004 mean AUROC for every model, the same as on the remaining datasets (at most 0.005). We also clarify that what is low-dimensional is the update of the replaced blocks, not the data. We agree with the reviewer that why pre-training produces this structure is not yet understood (as stated in App. B.3) and left for future work.

... serves as a rough proxy; i would not assume it to be a "consistent predictor" - but I'm open for references justifying it. (Q2)

We agree that this gap is a rough proxy, and now refer to it as such in the paper. With "consistent" we only meant that its correlation with the cost has the same sign and is significant for all eight models. Silvey and Liu [1] use the same kind of gap: they define the non-linearity of a dataset as the AUC difference between XGBoost and a logistic regression. It is also related to measuring the linear separability of a classification problem by the error of a linear classifier [2]. Our analysis also includes a second proxy that involves no linear model, the depth of the tuned GBDT. Datasets that need deeper trees also lose more, significantly for seven of the eight models at half depth and with one block kept. We added this result in the revised pdf (App. B.3).

... the performance is rather model-specific than a claim that holds in general. (Q3)

At half depth and on six of the eight datasets, all four models lose at most 0.02 AUROC relative to their full version at every magnitude of the shift. The larger average gap of the dual-axis models comes from the remaining two datasets (SUPPORT2 and Thyroid). We agree that these models are less robust there, which is why Table 19 reports each model separately.

... for any deployment settings one might need a fallback to a re-calibration. (Q3)

Yes, when the feature–target relation changes, the map should be refitted, in particular for the dual-axis models. But this needs no additional labels and takes seconds (Table 32). In this case, the full model also needs a new support set to learn the new relation in context and the refit only adds one forward pass and a closed-form fit on this new context.

... completely new feature-target interactions where some spurious features might get more important and vice versa. (Q3)

Following the reviewer's suggestion, we ran two new experiments with the same setting of Table 19, and in both of them a refit with no additional labels is again sufficient. In the first, the feature with the weakest effect on the source becomes the most important one and the strongest one loses its effect. In the second, an interaction between the two strongest features, absent on the source, is added to the relation. At half depth, a map reused from the source costs the single-axis models at most 0.006 AUROC and the dual-axis ones up to 0.040 (TabPFNv2, when the two features exchange their roles). After the refit, every model is within 0.005 of its gap without shift, also with one block kept. We added this result in the revised pdf (App. B.12).

We thank the reviewer again for the constructive discussion that helped improving the paper and remain available for further clarifications.


[1] Silvey, Liu. Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study. Journal of Medical Internet Research, 2024.

[2] Ho, Basu. Complexity Measures of Supervised Classification Problems. IEEE TPAMI, 2002.

Reviewer_qzH1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether pretrained in-context tabular foundation models require their full transformer depth during inference. It replaces a contiguous block window with an affine translator fitted by ridge regression to cached hidden states. Classification windows are selected using teacher-to-compressed-model KL divergence without calibration labels; dual-axis architectures receive feature-specific translators. Across eight models and 171 classification datasets in total, the authors report approximately 0.002 or less median AUROC degradation when replacing half the blocks. More aggressive compression produces architecture-dependent losses, while regression generally requires more depth. Additional experiments examine calibration budgets, cross-dataset transfer, context length, hardware efficiency, and shallow pretraining using nanoTabPFN. The paper presents both a post-training compression procedure and an empirical analysis of where existing transformer stacks are replaceable.

Critical strengths
  1. Broad evidence for the central compression finding. The paired full-versus-compressed evaluations cover multiple architectural families, three seeds, and diverse datasets. The particularly consistent Half-depth results in Figure 5 and Tables 9–14 constitute a meaningful empirical finding. Appendix A documents coverage differences, support caps, preprocessing, and ensemble settings, allowing the results to be interpreted within a reasonably explicit protocol.

  2. Useful interventions and component analyses. The identity baseline demonstrates that successful compression requires more than simply deleting inactive layers. The per-feature/shared comparison, window-selection ablations, calibration-budget sweep, and gradient-refinement experiments address substantive alternative explanations (Appendix A.3; Tables 4, 6–8). These make the study more informative than representation-similarity measurements alone.

  3. The paper investigates important failure conditions. Regression, long contexts, cross-dataset transfer, and clinical threshold/subgroup analyses expose limitations that aggregate classification AUROC could conceal (Appendices B.10, B.13–B.15). The native-shallow and compressed-topology controls in Section 7 also provide useful preliminary evidence, with their limited scale appropriately acknowledged.

  4. The contribution remains useful despite modest algorithmic novelty. Closed-form block approximation already appears in TOAST and ReplaceMe, while TACTICL studies task-aware TFM compression. Section 2 acknowledges these connections. The present contribution is primarily the breadth of the TFM audit, the classification calibration procedure, and the analysis of architectural differences—not the invention of linear block replacement. The cached/re-encode and stack/end-to-end efficiency measurements further strengthen its practical relevance.

Critical weaknesses
  1. Some mechanistic conclusions exceed what the interventions identify. Section 4 and the conclusion state that non-affine computation resides in a single critical dual-axis block; Section 5.1 claims to isolate where core computation occurs. However, the translator is fitted using target-conditioned teacher activations, and the encoders remain intact. Table 17 also shows that preserving TabPFNv2’s individually critical first block performs substantially worse than preserving its last block. The authors explain this through interface and contiguous-window constraints, but that explanation itself illustrates why replacement sensitivity is not a direct measure of intrinsic computational necessity. Similarly, stable attention entropy in Figure 10/Table 5 does not establish unchanged attention allocations.

  2. The early-exit comparison does not isolate representation translation from access to teacher supervision. Appendix B.11 trains logistic regression on support labels, whereas the translator receives full-model hidden-state targets and uses teacher predictions for selection. Consequently, its reported advantage in 81% of configurations changes both the architecture and the supervision source. This does not establish that preserving the pretrained head and matching hidden states is necessary, relative to a readout distilled from the same teacher.

  3. Section 3 requires a full teacher pass, candidate-specific fitting, and evaluation through retained suffixes. Appendix A.3 reports 33–109 ms per window, but Tables 1 and 23–24 do not establish aggregate initialization latency, peak setup memory, or amortization over subsequent queries. This matters especially when end-to-end re-encode improvements are only 1.0–1.4×. Table 24 additionally reports a 75% full-model stack-time share for TabPFNv2, an 11.2× stack speedup, and a 4.63× end-to-end speedup. With unchanged outside-stack cost, the reported decomposition predicts $ \frac{1}{0.25+0.75/11.2}\approx3.15\times $ Absorption, timing boundaries, or other changes could explain the difference; it should not be treated as proof of incorrect measurements.

Questions for authors

Q1. Can the localization claims be restricted to sensitivity under the proposed affine replacement? If stronger claims are intended, what evidence separates indispensable computation from head-interface alignment and information captured by the task-specific translator?

Q2. How does the translator compare with a teacher-logit-distilled linear readout using the same representations, calibration positions, and selection budget? This would directly test the interpretation attached to Table 16.

Q3. What are the complete calibration/search latency and peak memory, including teacher extraction and candidate scoring? Please also explain the TabPFNv2 stack/end-to-end timing relationship in Table 24 and specify whether calibration queries are capped during KL evaluation.

2 comments

We thank the reviewer for the careful and constructive review and for finding the consistent half-depth results "a meaningful empirical finding". We are also grateful that the reviewer considers our interventions to "address substantive alternative explanations". We address each point below, where section and table numbers refer to the revised PDF.

Can the localization claims be restricted to sensitivity under the proposed affine replacement? (W1, Q1)

Yes, and we thank the reviewer for this precise observation. We agree that our interventions measure sensitivity under the affine replacement and we restricted the statements in the revised pdf.

... stable attention entropy in Figure 10/Table 5 does not establish unchanged attention allocations. (W1)

We agree that entropy measures how spread out attention is, not which rows receive it. Sec. 4 and App. B.1 (Figure 10, Table 6) are updated accordingly.

How does the translator compare with a teacher-logit-distilled linear readout using the same representations, calibration positions, and selection budget? (W2, Q2)

We thank the reviewer for this insightful suggestion. We agree that the original comparison changed both the readout and the source of supervision, and we ran the proposed experiment (App. B.13, Table 21). On the same hidden states, we compared our map with two logistic readouts fitted on the same adaptation rows (the calibration rows of the submission): a label probe, and a distilled readout trained to match the full model's predicted probabilities instead of the labels. All exit depths are evaluated, so no window is selected, and the readouts use a fixed regularization (C = 1).

As the reviewer anticipated, the distilled readout performs similarly to our map. Over all configurations (seven models, eight clinical datasets, three seeds, every exit depth), our map matches or beats it in 46% of them (51% when our map, which also uses support rows, is fitted on the adaptation rows alone), and their pooled medians agree within 0.001 at early, middle and late exits. Our map still matches or beats the label probe in 88% of the configurations (81% of 2,664 in the submission: that table still used an earlier fit of our map without the ridge penalty for most configurations, and also counted the exit after the last block). This advantage therefore comes from fitting to the full model rather than from preserving the pre-trained head, and we reworded the introduction, Sec. 5.1, App. B.13 and the conclusion accordingly. However, our method remains the more general tool: it replaces any window of blocks, applies unchanged to regressors and, for the single-axis models, can be reused or pooled across datasets.

What are the complete calibration/search latency and peak memory, including teacher extraction and candidate scoring? (W3, Q3)

We now report the complete one-off setup cost for all eight models on MIMIC-III (Sec. 6, App. B.19, Table 32). It includes the forward pass that extracts the hidden states, the full model's predictions on the adaptation rows, and the ridge fit and KL score of every candidate window. At half depth, the setup takes 0.8–8.4 s with the window search and needs no more GPU memory than one prediction call, except for TabDPT (913 vs. 624 MB). With the context re-encoded on every call, the search is amortized after 50–391 calls of 512 queries. For the single-axis models, the search can be skipped at half depth, since a fixed window performs about as well on average (Table 9): the setup then takes 0.1–0.7 s. It is amortized after 12–35 calls for the five models whose end-to-end speed-up is at most 1.2×, and after 241 calls for TabDPT, whose full call takes only 7.9 ms.

... specify whether calibration queries are capped during KL evaluation. (Q3)

Yes, they are capped: the adaptation split is subsampled to at most 500 rows, and every candidate window is scored on these same rows, which keeps the cost of the search bounded on large datasets. Sec. 3 now states this.

Please also explain the TabPFNv2 stack/end-to-end timing relationship in Table 24 ... (W3, Q3)

We thank the reviewer for this careful check. We run TabPFNv2 through its official Python package (tabpfn 2.0.9, from Prior Labs), and the difference comes from how the package handles the weights: on its default path, it copies the model weights to the GPU and back on every prediction call, and our timing counted this copy as time outside the stack. Since the copy shrinks with the kept weights (from about 7 ms with all blocks to under 2 ms with one block kept), removing blocks also shortens the time outside the stack, which largely explains why the end-to-end speed-up exceeds what the stack share alone predicts. The caption of Table 31 (Table 24 in the submission) now explains this.

We hope with these clarifications have addressed all of the reviewer's comments and remain available for further clarifications.

Thank you to the authors for the detailed explanations. After carefully reading your response, I have no further questions. I will maintain my original score.

Reviewer_LNpzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the removal of consecutive blocks in tabular foundation models (TFMs), replacing them by an affine layer fitted on an unlabeled calibration set with regularization tuned on a validation set. This means that the removed blocks are still needed for fitting the affine layer, but can be dropped for prediction/inference on the test set. The paper argues that dropping half of the (in-context) blocks only incurs a small performance loss, while for dropping all but one block the picture depends on the TFM and is worse for regression. Experiments quantify the importance of different layers and whether similar benefits can be achieved with from-scratch training of smaller models.

Critical strengths

The paper has a clear scope, is rather easy to read and understand, contains useful ablations, and the main idea is simple. Its general topic is relevant.

Critical weaknesses

For practical use, the method may be less attractive than it initially sounds. The paper argues that the losses from dropping blocks are often small, but it is unclear in the main paper what the relative magnitude of these losses are compared to other learning methods. (I see that there is some comparison to XGBoost and LightGBM in the appendix but IMO it should be in the main paper.) If the average AUROC on the datasets is 0.95, losing 0.01 could mean a 20% increase in 1-AUROC, which could be the difference between a top TFM and random forest (judging by improvability differences on TabArena). In this case, dropping most of the ICL layers for a 2.4x inference speedup (Figure 1) wouldn't be very attractive. (The method makes fit time a bit slower but probably <2x.) I suspect that reducing n_estimators provides a better tradoff unless we are already at a single estimator like in the paper. Additionally, subsampling the training set could provide another axis of speedup. In my opinion, it would be good to compare the time-performance tradeoff with these two. (Full fine-tuning of the reduced-block model would also be another idea to trade predict for fit time, but it's not a standard method.) Moreover, by the paper's own ablations, the method would be rendered basically useless if model developers would just train a model with fewer layers (though the method can be useful for now since they don't do it).

The appendix (l. 1133) mentiones that support sets are capped at 3K rows. I didn't find this mentioned in the main paper. Why is this done? The small-data regime is probably the regime where a speedup is least useful, and it could also be that small datasets need fewer layers compared to large datasets.

In Figure 1, I think it could be useful to show results on more than one dataset - otherwise it is unclear whether that dataset is just very easy or very noisy.

Questions for authors
  • Have the authors tried whether using a MLP adapter (perhaps residual) instead of an affine one can improve the results? After all, the skipped layers also contain MLPs.

Minor comments:

  • l. 321: feature work -> future work
  • Figure 7: The difference between black circles and black squares is hard to recognize.
  • I find the use of the name "calibration" split a bit misleading, as calibration already has another meaning.
4 comments

We thank the reviewer for the valuable feedback, and for recognizing that the paper "has a clear scope" and "contains useful ablations". We are also grateful that the reviewer finds that "the main idea is simple" and that its "general topic is relevant". We address each point below, where section and table numbers refer to the revised PDF.

... it is unclear in the main paper what the relative magnitude of these losses are compared to other learning methods. (W1)

We understand the reviewer's concern about the relative magnitude of the loss, and we agree that a raw ΔAUROC can hide a large relative change in error. Table 13 checks whether compression meaningfully weakens TFM performance and at half depth it does not: when the full and the half-depth models are ranked together over the suite datasets, the half-depth models keep the same order as the full ones, and half-depth TabFM is outranked only by the full TabFM. Following the reviewer's suggestion, we now also report the comparison with tuned GBDTs in the main text (Sec. 5.1): at half depth, TabFM performs slightly better than tuned LightGBM and XGBoost (mean AUROC 0.897 vs. 0.888), while TabICL and TabPFN-3 perform about the same (0.888 and 0.886) and still rank above both (average rank 3.88 and 4.61 vs. 5.29 and 5.44; Table 13), even though the GBDTs are trained on the whole training split.

If the average AUROC on the datasets is 0.95, losing 0.01 could mean a 20% increase in 1-AUROC, which could be the difference between a top TFM and random forest ... (W1)

Following this reasoning, we measured the relative increase in error on every dataset (i.e., the loss in AUROC divided by 1 − AUROC of the full model) and report it in a new analysis (App. B.8, Table 15), summarized in Sec. 5.1. At half depth, the median increase ranges from 0.6% to 5.4% across models, a fraction of the roughly 20% gap between a default random forest and the top TFMs on TabArena (computed with the same formula). In the setting of Figure 1, TabPFN-3 with a single block kept, the median increase is 7.8%, less than half of this gap, and it stays at 5–14% for the two Orion models. The other models need more depth in this extreme setting, which is why we recommend half depth for deployment.

I suspect that reducing n_estimators provides a better tradoff unless we are already at a single estimator like in the paper. (W1)

We added the comparisons with fewer estimators and with a subsampled training set in new appendix sections (App. B.20 and B.21). As the reviewer noted, seven of the eight models already run with a single member in the paper, so for them this reduction is already applied and depth is the remaining lever. For TabICL, which uses eight members by default, we agree that reducing the number of members is the larger lever: a single member is 6.8× faster at a cost of 0.004 AUROC. The two reductions can be combined: TabICL with one member at half depth is 7.5× faster (Table 34).

Additionally, subsampling the training set could provide another axis of speedup. (W1)

We measured the accuracy and end-to-end GPU latency of the full model with fewer support rows on MIMIC-III (App. B.21, Table 37). For the three models where the transformer stack dominates runtime (TabPFNv2, TabDPT, and LimiX-16M), half depth is more accurate than the full model with fewer rows at the same speed: it changes AUROC by +0.001, −0.004, and −0.004, compared to −0.021, −0.028, and −0.015. Stronger subsampling is faster but quickly hurts accuracy: with 1,000 rows instead of 3,000, seven of the eight models lose 0.021–0.040 AUROC, compared to at most 0.008 at half depth. This is not specific to MIMIC-III: across MIMIC-III, eICU, Covertype, and Higgs, half depth with 3,000 rows is more accurate than the full model with 1,000 rows in 27 of the 32 model–dataset pairs.

... the method would be rendered basically useless if model developers would just train a model with fewer layers (though the method can be useful for now since they don't do it). (W1)

We agree with the reviewer that models pre-trained with fewer layers would offer similar savings, and our nanoTabPFN study in Sec. 7 points in this direction. However, this requires pre-training from scratch, and, to the best of our knowledge, most open-weight TFMs are released in a single size, with 12 to 24 blocks. Our method is, and will remain, useful for obtaining TFMs at operating points other than the released sizes: it brings these savings to any model with a single forward pass and a closed-form fit that needs no labels. It is also orthogonal to a smaller pre-trained model, since the map can be applied to it as well. Fine-tuning the reduced model, which the reviewer also mentions, could recover accuracy at the cost of extra fit time, but it requires labels and gradient updates, so it falls outside the label-free setting we study.

The appendix (l. 1133) mentiones that support sets are capped at 3K rows. ... Why is this done? The small-data regime is probably the regime where a speedup is least useful, and it could also be that small datasets need fewer layers compared to large datasets. (W2)

We capped the support set at 3,000 rows only to keep our exhaustive sweep, over every window on all 171 datasets, within the memory of one GPU. It is not a limit of the method and Sec. 4 now states the cap explicitly. To verify that the cap does not drive our conclusions, in a new experiment we re-ran all suite datasets with more than 3,000 training rows using up to 10,000 support rows (App. B.15, Table 24). At half depth, the compressed models benefit from the extra rows almost as much as the full models: their cost goes from −0.004 to −0.006 AUROC on average and stays within 0.011 for every model, also on larger datasets, where a speed-up is most useful. With one block kept, the results confirm the reviewer's intuition that larger datasets need more layers: the compressed models gain little from the extra rows, so the gap to the full model grows, most for TabDPT and LimiX-16M, while TabPFN-3 and Orion-Bix are barely affected.

In Figure 1, I think it could be useful to show results on more than one dataset - otherwise it is unclear whether that dataset is just very easy or very noisy. (W3)

We added a new figure in the appendix (Fig. 12), referenced from the caption of Figure 1, that shows the median and interquartile range of TabPFN-3 over all 171 classification datasets, next to the MIMIC-III curve of Figure 1. From half depth on, the model loses slightly more AUROC on MIMIC-III than on the median dataset, so MIMIC-III is not a favourable example.

Have the authors tried whether using a MLP adapter (perhaps residual) instead of an affine one can improve the results? (Q1)

We thank the reviewer for this interesting question. Following it, in a new experiment (App. A.3, Table 5) we tested the use of two MLPs instead of the affine map, both trained without labels to match the output of the replaced blocks: a residual MLP added on top of the affine map, and a plain MLP that replaces it. Averaged over seven models and eleven datasets, neither MLP helps at half depth: the residual MLP gives the same cost as the affine map (−0.002 mean AUROC), and the plain MLP a larger one (−0.004). With one block kept, the residual MLP helps (from −0.016 to −0.012), mostly for TabPFNv2, but it needs gradient training and, for the single-axis models, four times the parameters of the affine map. We have added this result to the Limitations section.

l. 321: feature work -> future work ... Figure 7: The difference between black circles and black squares is hard to recognize. (Minor)

We thank the reviewer for spotting these. We fixed the typo, and we changed the markers in Fig. 7.

I find the use of the name "calibration" split a bit misleading, as calibration already has another meaning. (Minor)

Following the reviewer's suggestion, we renamed the "calibration split" to "adaptation split" throughout the paper to avoid confusion.

We hope the additional experiments address the reviewer's concerns and we remain available for further clarifications.

Thank you for the detailed response. It resolves my concerns to some extent, though my overall impression remains that the direct impact of the proposed method is limited to moderate. Therefore I raise my score to 3 but not 4.

Minor comment: You wrote "Fine-tuning the reduced model, which the reviewer also mentions, could recover accuracy at the cost of extra fit time, but it requires labels and gradient updates, so it falls outside the label-free setting we study." - I thought you use labels to tune the regularization hyperparameter?

We thank the reviewer once again for the useful feedback and for increasing the score. Regarding the question on the regularization hyperparameter λ\lambda, we select it without labels (App B.5). We hold out 25% of the rows used for the fit and choose the λ\lambda whose map best reconstructs the hidden states after the replaced blocks on these rows (the fitting residual of Sec. 3); the map is then refitted on all of them. This criterion compares hidden states only, so the "validation split" of Sec. 3 needs no labels. We updated the manuscript and made this explicit also in Sec. 3 of the main paper.

Reviewer_v1GjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether tabular foundation models (TFMs) can be made shallower without retraining. It replaces a contiguous window of transformer blocks with a closed-form affine map fitted by ridge regression on unlabeled hidden states from the support and calibration rows. For models that attend across both rows and features, a separate map is fitted for each feature. The replacement window is selected by minimizing the output KL divergence between the compressed and full models on a calibration split.

Across eight TFMs from six model families and 171 classification datasets, replacing half the transformer blocks incurs a median AUROC loss of at most 0.002, while several models remain competitive even with only one block retained. In regression, half-depth replacement is also nearly lossless, although retaining only one block substantially degrades most models. The authors provide several controls: dropping blocks without the affine map causes performance to collapse, the label-free translator matches or outperforms a supervised early-exit probe in 81% of configurations, and repeatedly applying the retained block does not recover comparable performance.

The paper also analyzes where different architectures perform their computations, measures GPU and CPU inference speed-ups and memory savings, evaluates whether fitted maps transfer across datasets, and compares post-training compression with training shallow models from scratch using nanoTabPFN.

Critical strengths
  1. Strong controls that make the results credible. The paper does more than just show that removing blocks preserves accuracy. Dropping the same blocks without the affine map causes most models to collapse toward chance (as shown in table 9), showing that the skipped blocks perform meaningful computations. A logistic-regression probe trained with labels on the representation before the replacement window is matched or outperformed by the label-free map in 81% of 2,664 configurations. This suggests that the result is not simply due to the class being linearly readable from the earlier representation. Repeatedly applying the retained block does not recover performance, and adding loops alongside the map provides no gain. Gradient-based refinement of the map changes AUROC by at most 0.003 at width 4. Together, these controls support the central finding that the affine map approximates the skipped computation by moving intermediate representations into the space expected by the remaining blocks and the original prediction head.

  2. A broad study that reveals where different architectures use their depth. The study covers eight TFMs from six model families, 171 classification datasets, 41 regression datasets, and three seeds. At half depth, the median AUROC loss is at most 0.002 for every model, and the ranking against tuned GBDTs remains unchanged. The paper also shows that representation similarity and attention patterns do not reliably predict which blocks can be replaced. Instead, the authors directly investigate computational redundancy by replacing individual blocks. Their findings show that single-axis models rely primarily on their first block, dual-axis models have a critical block, and TabPFN-3 relies substantially on its encoders. The method is also straightforward to apply: it uses a closed-form ridge fit on unlabeled hidden states, takes approximately 33–109 ms per window, and requires no gradient updates. In some models, the fitted map can be absorbed into the encoder or prediction head. This makes the paper valuable not only as a compression method but also as a systematic study of how tabular foundation models use depth.

Critical weaknesses
  1. The title claims more than per-task fitting establishes. The affine map and replacement window are fitted separately for each dataset. This shows that, for a given task, a window of transformer blocks can be replaced by a linear map fitted to that task. It does not establish that the models are intrinsically shallow.
    • Dropping the blocks without the map collapses most models, showing that the skipped blocks perform meaningful computations.
    • Maps transfer poorly across schemas. At half depth, a borrowed map retains 60–65% of a local map’s benefit for TabICL and TabPFN-3, about 30% for TabDPT, and none for TabFM.
    • In nanoTabPFN, maps fitted on prior tasks achieve only 0.43–0.83 AUROC, compared with 0.85–0.90 for maps fitted on the target task.
    • Depth also matters under extreme compression: dual-axis models lose 0.05–0.10 mean AUROC, and most regressors lose 0.10–0.57 median R² when only one block is retained. The evidence for native shallowness is limited to a 1.4-million-parameter nanoTabPFN model. A more precise claim would be that substantial portions of pretrained TFM stacks can often be replaced by task-fitted affine maps, rather than that TFMs are generally “effectively shallow.”
  2. Practical savings are modest at the recommended setting and require a full-model pass for each new task. At half depth, the configuration recommended by the authors for deployment, end-to-end GPU speed-ups range from 1.0× to 1.8× when the context is re-encoded (Table 1). The abstract’s headline of “up to 6×” comes from the more aggressive one-block LimiX setting, which incurs a mean AUROC drop of 0.052. Moreover, fitting the map and selecting the replacement window require hidden states obtained from a full-model forward pass on the target support set and unlabeled calibration queries (Section 3). Because maps transfer poorly across schemas, the paper does not establish that this cost can generally be avoided by precomputing a compressed model. The savings therefore depend on amortizing the initial full-model pass and compression procedure over many subsequent queries on the same table. This weakens the resource-constrained deployment motivation when the full model must still run on the target hardware for every new task. There is also a parameter-efficiency limitation for dual-axis models on wide datasets: the per-feature map can exceed the parameters it replaces in 12 of 14 datasets with more than 100 columns. Finally, the paper does not compare the method against other compression approaches or smaller pretrained TFMs at matched accuracy, latency, or memory budgets. Such comparisons would help establish whether the proposed method offers a practical advantage over simpler alternatives.
Questions for authors
  1. What does compression cost for a new task, and when does it pay off? For a new task with Q queries, could you compare the total cost of the full pipeline (full-model pass, window search, map fit, compressed inference) with running the full model on all queries? At what Q does compression win? How does this fit the resource-constrained deployment motivation, given the full model must still run once per task?
  2. Does a fixed, dataset-agnostic compressed model work? Could you evaluate one fixed map and window per architecture, chosen on pooled or synthetic data, across the 171 datasets without fitting per dataset?
  3. How does it compare with alternatives at matched cost? At matched latency or memory, how does this method compare with TACTICL, a looped single-layer model, context compression, or a smaller pretrained TFM? Does the half-depth result also hold under each model’s default prediction-time ensembling?
3 comments

We thank the reviewer for the careful and constructive review, and for recognizing the "strong controls that make the results credible". We are also grateful that the reviewer finds the paper "valuable not only as a compression method but also as a systematic study of how tabular foundation models use depth". We address each point below, where section and table numbers refer to the revised PDF.

The affine map and replacement window are fitted separately for each dataset. ... It does not establish that the models are intrinsically shallow. (W1)

We agree that our results establish per-task replaceability and not intrinsic shallowness. We clarified this in the revised manuscript, which now states the claim per task in the abstract, the Introduction and the Conclusion. The results the reviewer lists on this point are already reported in the paper (Sec. 5, Sec. 7 and App. B.16) and we now also state the transfer results in the Limitations.

Does a fixed, dataset-agnostic compressed model work? (W1, Q2)

Following the reviewer's suggestion, we added two new experiments (App. B.16, Table 26), which show that a fixed map works as a default for the single-axis models at half depth, although fitting per task remains more accurate. In both, one map per architecture replaces the last half of the blocks of the six single-axis models and is applied unchanged to every dataset the model runs on (155–171 datasets). The dual-axis models are not included, since their per-feature map is tied to the columns of one dataset. In the first experiment, the map is fitted on the pooled hidden states of all other datasets and applied to the held-out dataset: it does at least as well as dropping the blocks on 70–96% of datasets and stays within 0.01 AUROC of the per-task map on 52–93% of them. In the second, we split the datasets into five groups, keeping all versions of a dataset in the same group, and apply to each group a map fitted on the other four, so the map never sees another version of the test dataset; the results are almost identical.

In both experiments, the mean cost of the fixed map is 0.007–0.025 AUROC across five of the six models, against 0.001–0.006 for the per-task map (for Orion-Bix, the mean change is +0.005, inflated by datasets where its full model is close to chance). At half depth, a map transferred from a synthetic source does no better than one from an unrelated real dataset for three of the four models (Table 25). The window can be fixed as well: for the single-axis models, always replacing the last half is within 0.003 mean AUROC of KL selection (Table 9), although KL selection leaves fewer datasets losing more than 0.01 and remains necessary for the dual-axis TabPFNv2.

The abstract's headline of "up to 6×" comes from the more aggressive one-block LimiX setting ... (W2)

We clarified this in the abstract, which now states that the 6× speed-up refers to the more aggressive setting with one block kept.

Practical savings are modest at the recommended setting ... (W2)

Beyond end-to-end GPU latency, half depth saves parameters and, with a cached context, query time and serving memory. It removes about half of the block parameters of the single-axis models (e.g., TabFM from 1,611M to 810M) and about 30% for the dual-axis models on MIMIC-III (Table 1). On very wide datasets, their per-feature map can outweigh the blocks it replaces, as we state in the Limitations (Sec. 8) and in App. A.3. With a cached context (TabPFNv2, TabPFN-3, TabICL), half depth makes queries 1.5–1.9× faster and reduces serving memory by 1.7–1.9× (4.3–10× with one block kept). For TabICL and TabFM, whose encoders limit the end-to-end speed-up to 1.0–1.2× at half depth, memory is the main gain (Sec. 6).

What does compression cost for a new task, and when does it pay off? ... At what Q does compression win? (W2, Q1)

We now report the complete one-off setup for all eight models on MIMIC-III (Sec. 6, App. B.19, Table 32): one full-model forward pass over the support and adaptation rows, the ridge fit, and either the KL window search or a window fixed in advance. At half depth, the setup takes 0.8–8.4 s with the search. For the six single-axis models, the window can be fixed (Table 9), and the setup then takes 0.1–0.7 s. With a fixed window, the setup is amortized after 16–31 single queries on a cached context (TabPFN-3 and TabICL), and after 12–35 calls of 512 queries when the context is re-encoded on every call (241 calls for TabDPT, whose full call takes only 7.9 ms). With the window search, the break-even rises to 115–379 single queries on a cached context (TabPFNv2, TabPFN-3 and TabICL) and to 50–391 calls over all eight models when the context is re-encoded, so for a small one-off dataset without caching, running the full model directly is cheaper.

How does this fit the resource-constrained deployment motivation, given the full model must still run once per task? (W2, Q1)

For the single-axis models, the fixed map and fixed last-half window from our answer to Q2 need no full-model pass on a new dataset, at a mean cost of 0.007–0.025 AUROC for five of the six models (Table 26). Otherwise, the full model runs only once per dataset and this can happen offline. Finally, the setup involves no training: beyond the support set that the model uses anyway, it needs at most 500 unlabeled adaptation rows.

At matched latency or memory, how does this method compare with TACTICL, a looped single-layer model, context compression, or a smaller pretrained TFM? (W2, Q3)

We address these in a new appendix section (App. B.21). TACTICL [1] is the concurrent work closest to ours, although the two are not strictly comparable: TACTICL trains an MLP for each removed block by gradient descent and uses labels to choose the blocks, whereas our map needs neither. At the same number of removed blocks, our map is comparable to TACTICL on TabPFN-3 and TabPFN-2.5 and more accurate on TabPFNv2, with higher prediction-time speed-ups on the same GPU and 0.5–2.5 s of setup against 63–93 s of search and MLP training (Tables 35 and 36). On MIMIC-III, we ran the authors' code on TabPFNv2 and, on TabPFN-3, a re-implementation that reuses their adapter and training code. With the same 500 adaptation rows for both methods, TabPFNv2 at half depth changes by +0.001 AUROC with our map and by −0.028 with TACTICL; when TACTICL is given 1,000 rows for each of its two steps (search and adapter training), it reaches −0.016. In TACTICL's own setting (TabPFN-2.5 on 34 TabArena tasks), with 6, 12 and 20 of the 24 blocks replaced, our map reaches median AUC ratios of 0.996, 0.9935 and 0.964, against at least 0.995, at least 0.995 and about 0.96 reported for TACTICL.

For context compression without training, we gave the full model the largest subset of its support rows at which it runs as fast as half depth, drawn at random or by class-stratified k-means [2] (Table 36). On the smallest datasets, even 10% of the rows is slower than half depth. On the remaining datasets, half depth is more accurate than random rows on TabPFN-2.5 (−0.007 vs. −0.014 mean AUROC), and k-means rows are no better than random ones.

The looped model [3] is pre-trained from scratch and do not compress the existing TFMs we study, so we do not compare with that method. However, it could be combined with our map.

A smaller TFM has to be pre-trained from scratch, whereas our map compresses an existing model without any training. Smaller versions are also rarely available. In language or vision, a model is typically released in many sizes (e.g., Qwen, or ViT from Tiny to Huge). To the best of our knowledge, most open-weight TFMs, including TabPFNv2, TabPFN-3, TabICL, TabDPT, TabFM and the Orion models, are instead released in a single size, and our map provides a smaller version of them. It can also be applied to a smaller model where one is available (e.g., new experiment on LimiX-2M; App. B.21). In our pre-training study, compressing a 12-block nanoTabPFN to k blocks matches pre-training a k-block model from scratch within 0.011 AUROC at every depth (Sec. 7, Fig. 7).

Does the half-depth result also hold under each model's default prediction-time ensembling? (Q3)

Yes. In a new experiment on the eight clinical datasets (App. B.20, Table 33), with every model at its default ensemble size, half depth changes the mean AUROC by −0.003 to +0.002, the same range as with a single member. TabICL already uses its default of eight members throughout the paper.

We hope with the following experiments and explanations we have addressed all of the reviewer's concerns and remain available for further clarifications.


[1] Koshil, Feurer, Eggensperger. TACTICL: Task-Aware Compression of Tabular ICL Models. AutoML, 2026.

[2] Feuer, Hegde, Cohen. Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks. arXiv:2311.10609, 2023.

[3] Balef, Koshil, Eggensperger. Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models. ICML, 2026.

Thank you for the detailed response and the new experiments. They answer my questions:

  • Setup cost (Q1). Table 32 gives the full one-off cost and the break-even point. With a fixed window, five of the six single-axis models recover the setup cost after 12–35 re-encoded calls, which addresses my deployment question.
  • Fixed map (Q2). The pooled per-architecture map is a useful addition. For the single-axis models at half depth, it stays within 0.010.01 AUROC of the per-task map on 52–93% of datasets, with no full-model pass on the new task.
  • Comparisons (Q3). The TACTICL comparison, the fewer-support-rows baseline and the default-ensembling check all cover the alternatives I asked about.

I will keep my positive rating.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback and for recognizing our "strong controls that make the results credible" (v1Gj), "clear scope" and "useful ablations" (LNpz), "broad evidence for the central compression finding" (qzH1), and "thorough analysis of the core claim" (cu6B). We are also glad that they found the paper "valuable not only as a compression method but also as a systematic study of how tabular foundation models use depth" (v1Gj), with efficiency measurements that "further strengthen its practical relevance" (qzH1). We reply to each reviewer individually and highlight the main additions below (in blue in the revised PDF):

We added comparisons against concurrent work TACTICL and against context compression, i.e., the full model with fewer support rows (Tables 35–37), together with the one-off cost of compressing a new dataset and the number of queries after which it pays off (Table 32) [v1Gj, LNpz, qzH1]. We also show that half-depth compression scales to 10,000 support rows (Table 24), works using each model's default ensembling (Table 33) [v1Gj, LNpz], and we analyze its OOD robustness (Table 19) [cu6B]. Finally, we evaluate the generalization of a single map across datasets (Table 26) and discuss why the map works and which dataset characteristics relate to when compression succeeds (Table 7) [v1Gj, cu6B].

We hope that these clarifications, together with the new experiments, address all the reviewers’ concerns and we remain available for any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.