- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TabMF, a method for learning dataset-level representations from pretrained tabular foundation models. The authors adapt TabPFN-2.5 and TabICLv2 by adding dataset-level embedding mechanisms tailored to each model's architecture, followed by contrastive training on synthetic datasets. The training objective encourages different views of the same dataset, obtained through row and feature subsampling and transformations, to produce similar embeddings. The learned representations are evaluated on several downstream tasks, including dataset retrieval, domain discrimination, dataset-property prediction, benchmark selection, and distribution-shift assessment.
Critical strengths
-
Novel approach to dataset representation learning. The paper presents an interesting way to repurpose pretrained tabular foundation models for dataset-level representation learning through contrastive post-training. Leveraging the knowledge already captured by TabPFN and TabICL is a promising direction that could support a range of downstream applications.
-
Strong empirical performance across multiple dataset-level tasks.
Critical weaknesses
-
Insufficient architectural ablation. The paper does not clearly separate the benefits of contrastive fine-tuning from those of the newly introduced dataset-embedding components. For example, comparing TabMF-PFN against a contrastively fine-tuned TabPFN using simple mean pooling, without the additional Column Transformer, would help determine whether the added architectural complexity is necessary for the reported improvements.
-
Unclear whether the embeddings capture predictive relationships or primarily dataset identity. The contrastive objective encourages different views of the same source dataset to have similar embeddings, but does not explicitly encourage similarity between independently generated datasets sharing the same feature–label relationship. A useful experiment would be to construct three datasets: A and B contain independently sampled observations with the same underlying relationship, , while C uses exactly the same feature observations as A but assigns labels according to a different relationship, . All three datasets would have comparable feature and label distributions. Ideally, the learned embeddings should satisfy:
This experiment would help determine whether TabMF captures similarities between prediction problems rather than primarily recognizing shared feature observations or different views of the same dataset.
Questions for authors
-
Contribution of the proposed dataset-level readouts (ablation request). Could the authors provide a controlled ablation separating the benefits of contrastive backbone fine-tuning from those of the newly introduced dataset-level readouts? Specifically, how would a contrastively fine-tuned TabPFN using simple mean pooling, without the additional Column Transformer, compare with TabMF-PFN under the same training protocol? This would help establish whether the additional architectural complexity is necessary for the reported improvements.
-
Dataset identity versus predictive relationships (additional validation). Does TabMF learn similarity between datasets sharing the same underlying feature–label relationship, or primarily recognize different views of the same dataset?
A useful controlled experiment would construct three datasets:
- A: Features with labels .
- B: Independently sampled features , with the same labeling rule as A.
- C: Exactly the same features as A, but with labels .
Using the same feature distribution and comparable class balances, one would expect representations that capture predictive-task similarity to satisfy
\{sim}(z_A,z_B)>\{sim}(z_A,z_C).Have the authors evaluated this distinction, or a similar controlled setting? Such evidence would strengthen the claim that TabMF learns general-purpose representations of supervised learning problems, rather than primarily dataset identity.
2 comments
Follow-up on rebuttal
Thank you again for your assessment of our paper and for suggesting the ablation and the A/B/C experiment, which we ran and reported in our response on 10 Nov. The ablation shows that the architectural additions improve results on both backbones. On TabICL, they raise retrieval from 0.830 to 0.938 and benchmark selection from 0.792 to 0.858. On TabPFN, they improve retrieval robustness, hard-margin separation, and benchmark selection, and they reduce epiplexity error. In the A/B/C test, TabMF-ICL and TabMF-PFN prefer the dataset with the same labeling rule in 99% and 91% of trials. We wanted to check whether these results address your concerns, and we would be glad to clarify anything further.
Thank you for your positive assessment and concrete suggestions. We conducted both requested experiments.
1. Contribution of the architectural additions.
We added contrastively fine-tuned mean-pooling baselines without our architectural additions, including the requested TabPFN variant without the Column Transformer. These comparisons measure the contribution of the added architectural components as a whole, including the projection MLP, relative to contrastively fine-tuned mean pooling.
TabPFN
TabICL
Retrieval: Recall@5; industry: tuned-SVM macro-OVR AUC; epiplexity: tuned kernel-ridge probe MAE (lower is better); selection: TabRepo ranking correlation averaged over k=2,3,4,5. Both probes follow the protocols in Appendix Tables 11 and 13. Bold denotes the best score per backbone and metric.
The architectural additions improve benchmark selection on both backbones. For TabPFN, fine-tuned mean pooling already recovers most of the overall retrieval gain; the additions further improve retrieval with 25% of features retained (0.890→0.937, p=0.012) and hard-margin separation (0.098→0.129, p<10−10). The additions also reduce epiplexity MAE from 0.211 to 0.179 on TabPFN (paired Wilcoxon, Holm-adjusted p=0.0006). They add negligible inference overhead (Appendix B.8). All trained variants share the prior, augmentations, and contrastive objective.
2. Predictive relationships versus shared observations.
We implemented your proposed A/B/C experiment: A and B have independent observations with labels y=1[x1+x2>0], while C reuses A's features with labels y=1[x1−x2>0]. Across 200 triples of 2,000-row datasets with independent standard-normal features and comparable class balances, the fraction satisfying sim(zA,zB)>sim(zA,zC), using cosine similarity, is:
Both TabMF encoders satisfy your proposed criterion: independently sampled datasets sharing a prediction rule are preferred over identical observations with a different rule in 99% and 91% of trials. The within-encoder differences sim(zA,zB)−sim(zA,zC) are significantly positive for both (one-sided Wilcoxon, p<10−27). This directly supports sensitivity to predictive relationships beyond shared observations.
Thank you again for suggesting these experiments; they helped clarify the architectural contributions and the representations’ sensitivity to predictive relationships. We will incorporate both into the revision. We hope these results address your concerns and would appreciate your updated assessment in light of this evidence.