acceptodds
Under review as a conference paper at ICLR 2027

TabMF: Learning Dataset Representations with Tabular Foundation Models

Abstract

Representations of supervised datasets support higher-level machine-learning tasks such as dataset retrieval and distribution-shift monitoring. However, existing approaches typically rely either on hand-crafted meta-features or on dataset encoders trained specifically for representation learning. We introduce TabMF, a framework that repurposes pretrained tabular foundation models (TFMs) as encoders of supervised datasets. TabMF equips a predictive TFM with an architecture-aware dataset-level readout and adapts the model end-to-end using contrastive post-training on stochastic views of synthetic supervised datasets. We instantiate TabMF on TabICLv2 and TabPFN-2.5 and evaluate the resulting representations across six tasks on unseen real datasets. Simple frozen TFM readouts yield competitive representations, indicating that predictive pretraining already induces substantial dataset-level structure. The two TabMF encoders obtain the strongest aggregate performance among the evaluated learned and classical dataset representations, with particularly strong results in dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Tabular Foundation Models, Dataset Representation Learning, Contrastive Learning, Dataset Embeddings, Tabular Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gB7WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TabMF, a method for learning dataset-level representations from pretrained tabular foundation models. The authors adapt TabPFN-2.5 and TabICLv2 by adding dataset-level embedding mechanisms tailored to each model's architecture, followed by contrastive training on synthetic datasets. The training objective encourages different views of the same dataset, obtained through row and feature subsampling and transformations, to produce similar embeddings. The learned representations are evaluated on several downstream tasks, including dataset retrieval, domain discrimination, dataset-property prediction, benchmark selection, and distribution-shift assessment.

Critical strengths
  1. Novel approach to dataset representation learning. The paper presents an interesting way to repurpose pretrained tabular foundation models for dataset-level representation learning through contrastive post-training. Leveraging the knowledge already captured by TabPFN and TabICL is a promising direction that could support a range of downstream applications.

  2. Strong empirical performance across multiple dataset-level tasks.

Critical weaknesses
  1. Insufficient architectural ablation. The paper does not clearly separate the benefits of contrastive fine-tuning from those of the newly introduced dataset-embedding components. For example, comparing TabMF-PFN against a contrastively fine-tuned TabPFN using simple mean pooling, without the additional Column Transformer, would help determine whether the added architectural complexity is necessary for the reported improvements.

  2. Unclear whether the embeddings capture predictive relationships or primarily dataset identity. The contrastive objective encourages different views of the same source dataset to have similar embeddings, but does not explicitly encourage similarity between independently generated datasets sharing the same feature–label relationship. A useful experiment would be to construct three datasets: A and B contain independently sampled observations with the same underlying relationship, y=1[x1+x2>0]y = \mathbb{1}[x_1+x_2>0], while C uses exactly the same feature observations as A but assigns labels according to a different relationship, y=1[x1−x2>0]y = \mathbb{1}[x_1-x_2>0]. All three datasets would have comparable feature and label distributions. Ideally, the learned embeddings should satisfy:

\{sim}(z_A,z_B) > \{sim}(z_A,z_C).

This experiment would help determine whether TabMF captures similarities between prediction problems rather than primarily recognizing shared feature observations or different views of the same dataset.

Questions for authors
  1. Contribution of the proposed dataset-level readouts (ablation request). Could the authors provide a controlled ablation separating the benefits of contrastive backbone fine-tuning from those of the newly introduced dataset-level readouts? Specifically, how would a contrastively fine-tuned TabPFN using simple mean pooling, without the additional Column Transformer, compare with TabMF-PFN under the same training protocol? This would help establish whether the additional architectural complexity is necessary for the reported improvements.

  2. Dataset identity versus predictive relationships (additional validation). Does TabMF learn similarity between datasets sharing the same underlying feature–label relationship, or primarily recognize different views of the same dataset?

A useful controlled experiment would construct three datasets:

  • A: Features XAX_A with labels y=1[x1+x2>0]y=\mathbb{1}[x_1+x_2>0].
  • B: Independently sampled features XBX_B, with the same labeling rule as A.
  • C: Exactly the same features XAX_A as A, but with labels y=1[x1−x2>0]y=\mathbb{1}[x_1-x_2>0].

Using the same feature distribution and comparable class balances, one would expect representations that capture predictive-task similarity to satisfy

\{sim}(z_A,z_B)>\{sim}(z_A,z_C).

Have the authors evaluated this distinction, or a similar controlled setting? Such evidence would strengthen the claim that TabMF learns general-purpose representations of supervised learning problems, rather than primarily dataset identity.

2 comments

Thank you for your positive assessment and concrete suggestions. We conducted both requested experiments.

1. Contribution of the architectural additions.

We added contrastively fine-tuned mean-pooling baselines without our architectural additions, including the requested TabPFN variant without the Column Transformer. These comparisons measure the contribution of the added architectural components as a whole, including the projection MLP, relative to contrastively fine-tuned mean pooling.

TabPFN

VariantRetrievalView AUCIndustryEpiplexity ↓Selection
Frozen, mean pooling0.8210.94140.6620.1770.845
Fine-tuned mean pooling, no architecture modifications0.9640.99800.6730.2110.854
TabMF-PFN0.9650.99850.6810.1790.867

TabICL

VariantRetrievalView AUCIndustryEpiplexity ↓Selection
Frozen, mean pooling0.8890.97750.6440.2030.594
Fine-tuned mean pooling, no architecture modifications0.8300.98260.6800.2570.792
TabMF-ICL0.9380.99810.7000.2220.858

Retrieval: Recall@5; industry: tuned-SVM macro-OVR AUC; epiplexity: tuned kernel-ridge probe MAE (lower is better); selection: TabRepo ranking correlation averaged over k=2,3,4,5k=2,3,4,5. Both probes follow the protocols in Appendix Tables 11 and 13. Bold denotes the best score per backbone and metric.

The architectural additions improve benchmark selection on both backbones. For TabPFN, fine-tuned mean pooling already recovers most of the overall retrieval gain; the additions further improve retrieval with 25% of features retained (0.890→0.9370.890\rightarrow0.937, p=0.012p=0.012) and hard-margin separation (0.098→0.1290.098\rightarrow0.129, p<10−10p < 10^{-10}). The additions also reduce epiplexity MAE from 0.211 to 0.179 on TabPFN (paired Wilcoxon, Holm-adjusted p=0.0006p=0.0006). They add negligible inference overhead (Appendix B.8). All trained variants share the prior, augmentations, and contrastive objective.

2. Predictive relationships versus shared observations.

We implemented your proposed A/B/CA/B/C experiment: AA and BB have independent observations with labels y=1[x1+x2>0]y=\mathbf{1}[x_1+x_2 > 0], while CC reuses AA's features with labels y=1[x1−x2>0]y=\mathbf{1}[x_1-x_2 > 0]. Across 200 triples of 2,000-row datasets with independent standard-normal features and comparable class balances, the fraction satisfying sim⁡(zA,zB)>sim⁡(zA,zC)\operatorname{sim}(z_A,z_B) > \operatorname{sim}(z_A,z_C), using cosine similarity, is:

EncoderPass rateEncoderPass rate
TabMF-ICL99%Frozen TabICL80%
TabMF-PFN91%Frozen TabPFN89%
PyMFE0%Dataset2Vec2.5%

Both TabMF encoders satisfy your proposed criterion: independently sampled datasets sharing a prediction rule are preferred over identical observations with a different rule in 99% and 91% of trials. The within-encoder differences sim⁡(zA,zB)−sim⁡(zA,zC)\operatorname{sim}(z_A,z_B)-\operatorname{sim}(z_A,z_C) are significantly positive for both (one-sided Wilcoxon, p<10−27p < 10^{-27}). This directly supports sensitivity to predictive relationships beyond shared observations.

Thank you again for suggesting these experiments; they helped clarify the architectural contributions and the representations’ sensitivity to predictive relationships. We will incorporate both into the revision. We hope these results address your concerns and would appreciate your updated assessment in light of this evidence.

AuthorsOpenReview ↗

Follow-up on rebuttal

Thank you again for your assessment of our paper and for suggesting the ablation and the A/B/C experiment, which we ran and reported in our response on 10 Nov. The ablation shows that the architectural additions improve results on both backbones. On TabICL, they raise retrieval from 0.830 to 0.938 and benchmark selection from 0.792 to 0.858. On TabPFN, they improve retrieval robustness, hard-margin separation, and benchmark selection, and they reduce epiplexity error. In the A/B/C test, TabMF-ICL and TabMF-PFN prefer the dataset with the same labeling rule in 99% and 91% of trials. We wanted to check whether these results address your concerns, and we would be glad to clarify anything further.

Reviewer_oZeEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose TabMF, a model built on top of pre-trained table foundation models (TabPFN, TabICL) trained on synthetic views generated from the TabICL pre-training dataset to learn embeddings that represent the entire dataset. The authors show that the pre-trained foundation models themselves already contain useful signal about the dataset, and that TabMF can improve upon this signal. The authors evaluate this approach on a large set of benchmark datasets commonly used for tabular ML tasks and show that TabMF improves upon the performance of the pre-trained foundation models.

Critical strengths

The idea is interesting and the authors show that the representations derived by table foundation models can be meaningfully improved for dataset matching task. The finding that pretraind table foundation models' internal representations contains useful signal is an interesting finding as well.

Critical weaknesses

The evaluation is quite limited. The main results are showing performance on datasets that are ML-task oriented, which represent a narrow subset of tabular datasets that exist. The authors should also consider looking into literature on tabular data matching tasks such as schema matching, table unionability search or entity matching for a more meaningful evaluation of the realistic usefulness of this method. The authors do include TEmBed results very briefly, but for the given motivation of dataset retrieval, a more thorough evaluation on this task would be more convincing. The authors also do acknowledge that this method is focused on "supervised dataset" retrieval, which I suppose is a limitation of the method. The evaluation can also be improved by considering the effects of different types of features (categorical, numerical) to better understand whether the limitations of tabular foundation models such as weaker performance on categorical-dominant datasets can be overcome by the training proposed in this paper.

Questions for authors
  • Are there any results about the trend in performance between datasets with more/less categorical features? how does TabMF perform / how does it compare against the pre-trained models?
  • In the case of TEmBed, how is TabMF converted to work with the label-less datasets? That is not made very clear in the paper.
  • While the task of "supervised table matching" is quite niche, the hyperparatmer transfer is indeed a good motivation. But the results for this in the main benchmark tables seem to be missing in the main text. Can the authors provide more details about this evaluation and or why it is not included?
7 comments
AuthorsOpenReview ↗

Response to Reviewer oZeE - Part 1/2

Thank you for the careful review. We expanded the evaluation to address your concerns about practical data-matching tasks, categorical features, label-less tables, and hyperparameter transfer.

1. Data-matching tasks.

TabMF's table-level vectors directly support table union search. We evaluate SANTOS-Small (50 queries, 550 lake tables; Khatiwada et al., 2023) using Starmie's evaluation code and the authors' Starmie and SANTOS implementations. All TFM encoders receive numerical values and within-table categorical codes, without headers or cell-text semantics.

We evaluate Starmie and SANTOS both with original cell text and with the same information available to TabMF: numerical values are unchanged, while categorical strings become table-local identifiers. This preserves category membership and frequencies within each column, but removes lexical meaning and shared category identities across tables. These input ablations test performance under TabMF's information constraints alongside the original-text comparisons. To handle label-less tables, we average normalized embeddings across up to five candidate target columns with 2–10 classes.

MethodMAP@10P@10R@10Input
Starmie (original text)0.9180.8200.612Cell text
TabMF-ICL0.8650.7920.581Numeric encoding
TabMF-PFN0.8310.7460.544Numeric encoding
SANTOS (original text, lake KB)0.8200.7600.567Cell text + lake KB
Sherlock†0.7820.6720.493Cell text
Pre-trained TabICL0.7490.6220.443Numeric encoding
Starmie (table-local categorical codes)0.7470.6100.444Numeric values + codes
Pre-trained TabPFN0.7360.6000.445Numeric encoding
PyMFE0.7270.6140.452Statistics
D³L, without column names†0.523—0.422Cell text + statistics
SANTOS (table-local categorical codes)0.0220.0220.017Numeric values + codes + lake KB

†From Fan et al. (VLDB 2023); all other results are our runs. Bold: best scores without cell-text semantics. Both Starmie rows use the same checkpoint trained on original text, without retraining on coded inputs. SANTOS rebuilds its lake KB for each input setting.

Both TabMF variants outperform their corresponding frozen backbones and PyMFE (paired bootstrap, p≤0.002p\leq0.002). TabMF-ICL also scores above SANTOS with original text and its lake-built KB (MAP@10 0.865 versus 0.820; p=0.31p=0.31), while Starmie with original text leads at 0.918.

Under the same input-information constraints as TabMF, Starmie's MAP@10 falls from 0.918 to 0.747, and both TabMF variants outperform it (p<0.001p < 0.001 for ICL; p=0.003p=0.003 for PFN). SANTOS falls from 0.820 to 0.022, showing its dependence on shared textual values for cross-table matching. Starmie and SANTOS were specifically developed for table union search, whereas TabMF learns general-purpose dataset representations and is applied here without task-specific post-training. In practice, headers and cell text are often missing, noisy, or anonymized; TabMF's values-only representations remain applicable in exactly these settings.

Schema matching.

On Valentine (544 table pairs), we embed each column as a one-column table with a constant target and no headers. We also evaluate four dedicated schema matchers using the authors' Valentine package.

MethodInputMRR ↑\uparrow
Similarity FloodingNames + schema0.729
TabMF-ICLValues0.700
Value overlap (Jaccard)Values0.700
Pre-trained TabICLValues0.667
Jaccard-LevenshteinValues0.593
TabMF-PFNValues0.584
Distribution-basedValues0.558
Pre-trained TabPFNValues0.483
CupidNames + types0.450

Both adapted encoders improve MRR over their frozen backbones. TabMF-ICL matches value overlap at the reported precision and significantly outperforms Distribution-based and Cupid (paired Wilcoxon, p<0.001p < 0.001). Similarity Flooding, which uses column names and schema structure, scores highest across all pairs. Its lead comes from the 90 pairs in which Valentine leaves column names unchanged, where name-based matchers rank every true match first. On the 450 pairs with altered names, TabMF-ICL is level with Similarity Flooding (MRR 0.699 vs. 0.673, p = 0.34) and significantly ahead of Cupid (+0.342) and Distribution-based (+0.143; both p < 0.001). These results extend TabMF's general-purpose dataset representations to column matching without headers or task-specific training, despite not being designed for column-level embedding.

We continue in Part 2 with the expanded TEmBed evaluation and answers to your questions Q1-Q3 on categorical features, label-less tables, and hyperparameter transfer.

AuthorsOpenReview ↗

Response to Reviewer oZeE - Part 2/2

2. A more thorough TEmBed evaluation.

We now evaluate all encoders on the three TEmBed tasks that use one vector per table. The remaining tasks require column-, row-, or cell-level representations, or natural-language queries.

EncoderSimilarity MAPShuffling accuracy: OriginalShuffling accuracy: OrderedType detection F1
HyTrel0.6750.9771.0000.681
TabMF-ICL0.6060.7321.0000.363
Pre-trained TabICL0.4970.7531.0000.324
TabMF-PFN0.4610.7731.0000.289
Pre-trained TabPFN0.4570.7941.000—
PyMFE0.4090.7051.0000.372
Dataset2Vec0.3500.4991.0000.285
Neural Statistician0.3490.6751.0000.182

Similarity: 16 GitTables datalakes; shuffling: 60 slices. Ordered: content-based column ordering at inference. Similarity MAP changes by at most 0.03 under this ordering.

TabMF-ICL outperforms every evaluated values-only encoder on similarity search (paired Wilcoxon, p≤0.002p\leq0.002). HyTrel, which reads headers and cell text, remains ahead on TEmBed (p=0.006p=0.006), consistent with its focus on semantic table understanding and pretraining on Wikipedia and Common Crawl tables. This strength does not extend consistently to our other evaluations: its Level-0 bootstrap-view hard margins are significantly below both adapted encoders (Holm p<1.6×10−16p < 1.6\times10^{-16}; Table 4). On dataset-property prediction, its domain AUC is 0.616, and epiplexity MAE is 0.3348, below both frozen backbones and Dataset2Vec on these metrics (Tables 11 and 13).

  • Similarity search. TabMF-ICL beats every values-only encoder on 13–16 of the 16 datalakes (paired Wilcoxon, p≤0.002p\le 0.002). HyTrel, which reads headers and cell text, remains ahead (−0.069-0.069, p=0.006p=0.006). The largest gap is on type detection, which is largely a semantic task.

  • Shuffling. The diagnostic identifies column-order sensitivity: all foundation-model encoders score ≥0.998\ge 0.998 when only rows are reordered. Applying a content-based column ordering at inference yields shuffling accuracy 1.00 for every evaluated encoder, with similarity MAP within 0.03 of the original.

Q1. Categorical vs. numerical features.

We split the 191 TabBench datasets by the share of categorical features (OpenML declarations) and compared each TabMF encoder with its pre-trained backbone.

Retrieval Recall@5Numerical (n=102n=102)(0,0.5](0,0.5] (n=34n=34)>0.5>0.5 (n=55n=55)
TabMF-ICL0.9180.9410.989
Pre-trained TabICL0.8850.8860.893
TabMF-PFN0.9510.9800.988
Pre-trained TabPFN0.8120.8120.809

Both TabMF encoders improve retrieval and hard margin over their frozen backbones in every feature-type bin (paired Wilcoxon, p≤0.015p\leq0.015). TabICL's retrieval gains increase with categorical share (Spearman ρ=0.21\rho=0.21, p=0.003p=0.003). No metric significantly worsens with categorical share.

Q2. Label-less tables in TEmBed.

TabMF represents supervised datasets (X,y)(X,y). For TEmBed, we assign a constant target (y=0y=0, one class), which supplies no feature–label relationship and lies outside the training distribution.

Q3. Hyperparameter transfer.

We expanded the HPO-B evaluation to all 935 tasks and enforced dataset-level source exclusion:

[Updated 17 Nov: the HPO-B pipeline did not standardize inputs to the two TabICLv2-based encoders, unlike our other evaluations. Corrected values: TabMF-ICL 0.232 (previously 0.12), pre-trained TabICL 0.248 (previously 0.274); all other encoders are unaffected.]

EncoderNeighbour quality ↑\uparrow
TabMF-PFN0.299
Pre-trained TabPFN0.286
PyMFE0.279
Pre-trained TabICL0.248
Size/shape meta-features0.242
TabMF-ICL0.232

Every representation transfers configurations significantly better than a random source, recovering 23–30% of the gap to the oracle source. TabMF-PFN scores above its frozen backbone and the meta-feature baselines, while TabMF-ICL scores lower. These differences are not statistically significant after Holm correction, so this evaluation does not establish a reliable ordering among these representations. The HPO evaluation was supplementary and did not contribute to the paper's six-task aggregate ranking.

Thank you again for the concrete suggestions; they motivated the new data-matching evaluations and helped clarify the scope of our representations. We hope these results address your concerns and would appreciate your reconsideration of the overall assessment in light of this additional evidence.

Thank you for your response. While they answer some of my questions, I am still not convinced of the usefulness of the current approach as it requires labels and is not able to take the semantic content of the table into context. I also suspect that one can do better than just assuming a single label for every row in TabEmbed, but I also suspect the lack of semantic input will be a bottleneck anyways. Also, the schema matching comparisons are missing modern baselines, like Magneto [1], SemStruct [2] or ISResMat [3]. I will maintain my score.

[1] Magneto: Combining Small and Large Language Models for Schema Matching, VLDB 2025

[2] SemStruct: Contextualizing Semantic Embeddings with Structural Information for Schema Matching, KDD 2026

[3] In Situ Neural Relational Schema Matcher, ICDE 2024

Reviewer_J4tgOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TabMF, a framework for learning dataset-level representations from pretrained tabular foundation models (TFMs). The method introduces architecture-aware readouts and contrastive post-training on synthetic datasets to generate transferable dataset embeddings. TabMF is implemented using TabICLv2 and TabPFN-2.5 and evaluated across six downstream tasks, including dataset retrieval, benchmark selection, and distribution-shift assessment. The results demonstrate competitive performance and reveal that pretrained TFMs already encode useful dataset-level information.

Critical strengths
  1. Comprehensive empirical evaluation. The authors evaluate the framework across multiple downstream tasks and compare it with classical meta-features, dedicated dataset encoders, and frozen TFM representations. However, these results demonstrate contribution of existing works.
Critical weaknesses
  1. TabMF’s dataset-representation formulation, view construction, and training strategy closely resemble Dataset2Vec. Its main substantive differences are the use of a predictively pretrained TFM and backbone-specific readouts. However, the difference with existing works are not fully discussed.
  2. Its main substantive differences are the use of a predictively pretrained TFM and backbone-specific readouts. However, the readout’s improves overall retrieval R@5 only from 0.964 to 0.965.The paper should therefore more clearly acknowledge the overlap with existing works, attribute the main gains to the real reason (e.g., contrastive adaptation proved by existing works), and avoid overclaming the contribution.
Questions for authors

see above

10 comments
AuthorsOpenReview ↗

Response to Reviewer J4tg - Part 1/3

Note: Parts 1–3 respond to the review as it stood before its revision, about ten hours before the end of the discussion period (rating 2 at the time). The current review lists different strengths and weaknesses; both versions are available in the review's revision history. We respond to the revised review in "Dataset2Vec: what the encoder can represent; the per-backbone role of the readout" and "Revised PDF addressing your final comment" in this thread.

Thank you for the detailed review. We ran new experiments for each of your concerns. We begin with the isolation of contributions, since it underlies several of the other points.

1. Isolating contrastive post-training from the readout.

We completed a 2×22\times2 ablation on both backbones: frozen versus contrastively fine-tuned backbone, crossed with mean pooling versus our added architectural components. In the frozen-backbone, learned-readout setting, only the added components are trained. All trained variants share the prior, augmentations, and contrastive objective. This separates the effects of backbone fine-tuning and the architectural additions, and examines their combination.

TabICL

CellRetrieval R@5R@5, 25% columnsHard marginSelection (k=2k=2–55)
Frozen, mean pooling (pre-trained)0.8890.6530.0070.594
Frozen, learned readout0.8880.7470.0680.811
Fine-tuned, mean pooling, no architectural modifications0.8300.6110.0590.792
Fine-tuned, learned readout (TabMF-ICL)0.9380.8840.1410.858

TabPFN

CellRetrieval R@5R@5, 25% columnsHard marginSelection (k=2k=2–55)
Frozen, mean pooling (pre-trained)0.8210.288−0.014-0.0140.845
Frozen, learned readout0.9390.8320.0240.776
Fine-tuned, mean pooling, no architectural modifications0.9640.8900.0980.854
Fine-tuned, learned readout (TabMF-PFN)0.9650.9370.1290.867

Paired tests over 191 TabBench datasets (Wilcoxon; McNemar for 25% columns) and 200 selection trials. Selection: TabRepo ranking correlation with K-means representatives, averaged over budgets k=2k=2–55 (per-kk curves in the appendix). Both fine-tuned mean-pooling cells have no projection MLP and were trained for 45k steps.

  • The two components contribute differently across backbones. On TabPFN, contrastive fine-tuning with plain mean pooling recovers nearly all of TabMF's overall retrieval improvement over the pre-trained model (0.821 →\rightarrow 0.964, p<10−26p < 10^{-26}). On TabICL, fine-tuning with mean pooling reduces retrieval performance (0.889 →\rightarrow 0.830), whereas combining fine-tuning with our architectural additions improves it to 0.938.

  • Training the readout on a frozen backbone is not enough. With the same readout, fine-tuning the backbone is significantly better than freezing it on every identity metric, for both backbones (p≤8⋅10−4p\le 8\cdot10^{-4}).

  • The architectural additions improve robustness and benchmark selection on both fine-tuned backbones. On TabICL, they improve overall retrieval (0.830 →\rightarrow 0.938) and benchmark selection (0.792 →\rightarrow 0.858). On TabPFN, overall retrieval is nearly unchanged (0.964 →\rightarrow 0.965), while the additions improve hard-margin separation (0.098 →\rightarrow 0.129, p<10−10p < 10^{-10}), retrieval with 25% of columns retained (0.890 →\rightarrow 0.937, p=0.012p=0.012), and benchmark selection (0.854 →\rightarrow 0.867).

We will add this analysis to the paper and describe the role of each component per backbone. (Added in the revised version, Section 5.8)

We continue in Part 2 with transfer across tasks and data sources, hyperparameter transfer, and computational cost.

AuthorsOpenReview ↗

Response to Reviewer J4tg - Part 2/3

2. Transfer across tasks and data sources.

Dataset-property prediction. The stronger nonlinear probes already reported in the appendix also support transfer beyond similarity tasks: TabMF-ICL improves domain AUC from 0.644 to 0.700, with significantly lower log-loss (Holm-adjusted p=0.0278p=0.0278; Table 11), while neither adapted encoder differs significantly from its frozen backbone on epiplexity MAE, despite lower rank correlations (Table 13).

Data beyond ML benchmarks. New evaluations extend the comparison to open-data and web tables:

Task / metricBackboneFrozenTabMF
SANTOS union search, MAP@10TabICL0.7490.865
TabPFN0.7360.831
GitTables similarity, MAPTabICL0.4970.606
Valentine schema matching, MRRTabICL0.6670.700
TabPFN0.4830.584

SANTOS uses the same averaged-target adapter for frozen and adapted encoders. Valentine embeds each column as a one-column table and is exploratory.

Both TabMF variants significantly improve retrieval over their frozen backbones across all three categorical-feature bins (paired Wilcoxon, p≤0.015p\leq0.015; see the per-bin results table in our response to Reviewer oZeE, Q1).

Hyperparameter transfer. We expanded HPO-B evaluation to all 935 tasks and enforced dataset-level source exclusion:

[Updated 17 Nov: the HPO-B pipeline did not standardize inputs to the two TabICLv2-based encoders, unlike our other evaluations. Corrected values: TabMF-ICL 0.232 (previously 0.12), pre-trained TabICL 0.248 (previously 0.274); all other encoders are unaffected.]

EncoderNeighbour quality ↑\uparrow
TabMF-PFN0.299
Pre-trained TabPFN0.286
PyMFE0.279
Pre-trained TabICL0.248
Size/shape meta-features0.242
TabMF-ICL0.232

Every representation transfers configurations significantly better than a random source, recovering 23–30% of the gap to the oracle source. TabMF-PFN scores numerically above its frozen backbone and the meta-feature baselines, while TabMF-ICL scores lower; these comparisons are not significant after Holm correction. This supplementary evaluation did not contribute to the six-task aggregate ranking.

3. Computational cost.

Earlier checkpoints quantify the performance obtained with smaller post-training budgets:

EncoderStepsGPU-hoursView AUCRetrieval R@5Selection (k=2k=2–55)
TabMF-ICL15k≈115\approx1150.99610.9270.759
TabMF-ICL (submitted)45k3150.99810.9380.858
TabMF-PFN20k≈117\approx1170.99830.9520.833
TabMF-PFN (submitted)45k2730.99850.9650.867

Post-training is a one-time cost, with negligible additional inference overhead (Appendix B.8). At 37% and 43% of the full compute, respectively, TabMF-ICL and TabMF-PFN already recover most of their view-discrimination and retrieval gains. Longer training primarily benefits benchmark selection, as shown above.

In Part 3, we address novelty and generalizability, including the choice of backbone families and the post-training prior.

AuthorsOpenReview ↗

Response to Reviewer J4tg - Part 3/3

4. Novelty.

Thank you for raising the question of methodological novelty. Our contribution is a framework for turning predictive TFMs into learned dataset-level encoders through architecture-aware readouts and contrastive post-training. The distinction from prior work is substantive: Regenwetter et al. (2026) mean-pool frozen TabPFN-2.5 activations for synthetic-data curation in engineering regression, whereas we explicitly learn dataset representations and evaluate their transfer across two TFM architectural families and up to 12 downstream tasks.

Our work differs on each of these points. We introduce (i) architecture-aware readouts, which extract dataset-level information according to how each TFM family encodes it rather than by generic pooling, and (ii) dataset-level contrastive post-training, which learns the representation instead of taking frozen activations as given. Both are developed and validated across two representative TFM architectural families and twelve downstream tasks.

These components are what drive the results. TabMF-PFN and TabMF-ICL obtain the two best aggregate positions across the six primary tasks (median ranks 1.5 and 2.0), ahead of every evaluated frozen backbone, dedicated dataset encoder, and classical meta-feature set (Table 1). Frozen readouts of the kind used in prior work are therefore not sufficient: the gain comes from the methodology we add.

Finally, our systematic evaluation yields two findings that are new to the literature: predictive pretraining alone induces useful dataset-level structure, and there is a trade-off between retrieval robustness and property recoverability.

5. Generalizability.

Backbones. The reviewer is right that we evaluate two backbones, and we acknowledge that the paper did not explain this choice clearly enough. The two models were selected as representatives of two architectural families, not as isolated instances, and we should have stated this explicitly. TabICLv2 compresses contextualized features into row representations before in-context attention, a design also underlying TabPFN-3. TabPFN-2.5 retains feature-group representations through alternating row- and feature-wise attention; LimiX and EXAONE Tabular likewise retain cell-level representations with attention across both axes. Kumo Tabular and Google's TabFM combine alternating row/column processing with row compression and in-context learning. Our architecture-aware readouts are therefore tested on two widely reused organizations of tabular context. We agree that this does not amount to evaluating every architecture, and we will revise the paper to make the family-level rationale explicit and to state this scope clearly.

Post-training prior. We agree that relying on a single synthetic prior is a legitimate concern, and we regret not having explained in the paper why we did so and what this prior contains. The choice reflects availability: among SOTA TFMs such as TabICLv2, TabPFN-2.5/3/3.5, Kumo Tabular, TabFM, LimiX, Seldon, MITRA, and EXAONE Tabular, only TabICLv2 publicly releases its pretraining-prior generator. The older public TabICL prior is the one alternative, but substituting it substantially reduces TabICLv2's predictive performance (Qu et al., 2026).

We also did not make clear how diverse this prior is. It samples varied graph structures and mixes neural networks, tree ensembles, Gaussian processes, linear and quadratic functions, discretization, clustering-inspired mechanisms, and products of functions. Post-training therefore covers multiple functional families and inductive biases, and all downstream evaluations measure transfer to unseen real datasets, none of which come from this prior. We will add this description and the rationale for the choice to the paper, and note the dependence on a single prior family as a limitation.

Limitations. Regression remains outside the evaluated scope (even if the proposed method could be seamlessly applied to a regression model since the readouts and objective do not depend on the prediction head, but a regression encoder requires its own post-training run) and is an interesting direction for future work. TableShift covers nine publicly available tasks; we report the observed correlations without claiming statistically established pairwise superiority (Appendix B.5).

Thank you again for the detailed suggestions; they motivated the additional ablations, transfer evaluations, and compute analysis. We hope these results address your concerns and would appreciate your reconsideration of the overall assessment in light of this additional evidence.

Comments on the paper

1 comment
Area_Chair_fXuJOpenReview ↗

Please Read Author Response

Dear Reviewers,

We're halfway through the discussion stage. The authors have provided their responses to your reviews. Please check whether their responses have addressed the concerns raised in your review. If not, please raise and leave some time for the authors to respond again. If your concerns have been resolved, please still respond to the authors. Let's keep the conversation going.

Many thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.