acceptodds
Under review as a conference paper at ICLR 2027

TabSQRT: Self-Query Representation Transfer for Tabular In-Context Learning

Abstract

Tabular foundation models (TFMs) predict query labels from labeled context examples; during training, the query labels themselves serve only as loss targets. We ask a different question: if a frozen TFM is allowed to see the answers, can it teach a student how to represent examples whose answers are hidden? TabSQRT (Self-Query Representation Transfer) turns this idea into supervision. TabSQRT constructs targets by supplying each labeled training block to a frozen TFM as both context and queries. A conditional diffusion student learns to generate the resulting decoder-input representations from standard in-context inputs, followed by classification post-training through the frozen decoder. In diagnostics with students trained separately on each dataset, larger target-construction blocks improve diffusion-student accuracy even as target decoding accuracy declines. This finding motivates assessing supervision through downstream student performance. Trained across 100,000 real-world tables, TabSQRT achieves 88.65% mean accuracy on 38 TabArena classification datasets without dataset-specific fine-tuning, compared with its LimiX teacher's 88.30%, and ties the highest reported mean accuracy among current frontier TFMs. Ablations support combining self-query targets, diffusion-based representation learning, and classification post-training for transfer to unseen tables.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Tabular Foundation Models, In-Context Learning, Representation Learning, Privileged Supervision, Conditional Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rZPKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a diffusion-based model for tabular prediction. It is trained by denoising noisy versions of the output target embeddings of a frozen teacher TFM. This kind of distillation is trained on datasets from TabLib. At inference time, it starts from random noise and denoises it toward embeddings that can be used for predictions. The denoising is conditional on the input's table encoding. Predictions are made with a post-trained output head. The idea is evaluated on the TabArena datasets with a custom validation protocol.

Critical strengths

The one critical strength of the paper is the combined idea of distillation and denoising to produce a generally suitable approach to obtain a diffusion-based model for tabular predictions. The idea is very interesting and leaves a lot of room for follow-up questions and research. The idea is worth sharing.

Critical weaknesses

I see two critical weaknesses: the work lacks quality and rigor. The presentation is generally confusing; the related work is minimal; several findings lack sufficient evidence; and the main validation protocol is flawed and provides insufficient positive evidence.

In short: in several places, the presentation lacks reasoning or detail (e.g., why pretrain on TabLib instead of a synthetic prior, why MotherNet is an alternative regime in the related work, how many datasets were filtered by the approach in Appendix A1). In its current form, the manuscript and its argumentative flow are unnecessarily hard to follow. Moreover, the authors claim to follow the official TabArena protocol but deviate significantly (wrong splits, wrong baselines, wrong metrics, wrong aggregation). As a result, the reported validation is extremely untrustworthy, and their methods' results across metrics and aggregation (unweighted vs rank) suggest bias from the self-chosen aggregation. Neither the deviation nor the new metric choices are justified. Moreover, they omit stronger TFM baselines, and their argument for improving the teacher is underdeveloped, especially since a basic fine-tuned version of Limix already seems competitive with the denoiser. A more rigorous evaluation and alternative post-training baselines are needed to justify the method based on predictive performance alone. Furthermore, the presentation needs significant improvement, including the results and the paper's argumentative flow.

Questions for authors

Given the paper's current state, there are no questions that would affect my assessment.

Reviewer_77HjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an approach for improving on tabular foundation models (TFMs) using a procedure that uses additional training data (from a corpus of tabular prediction tasks) in combination with an existing frozen TFM to train a "denoiser transformer" that modifies existing representations before decoding into a prediction target.

Critical strengths

I appreciate the authors showing multiple perspectives on comparing performance across baselines and their approach, and they seem to include a sufficient set of baselines. E.g., I appreciated the showcasing of pairwise win/loss rates, the accuracy-runtime trade-off, and the full set of results in Table 1.

Critical weaknesses

TL;DR: I don't see any clear intuition/motivation for the proposed method, and it doesn't appear to work very well in any case!

I see two major weaknesses: First, a lack of clarity / intuition around what signal the proposed method is intended to exploit. Second, the experimental results are underwhelming, showing little if any gain over existing TFMs, without further training.

To the first point, I had some difficulty understanding the idea, even after reading the abstract/intro/Figure 1. The intro outlines how TFMs work, and then jumps into what seems like a non-sequitur "This suggests a question...how should a frozen TFM construct label-informed representation targets that remain useful to a student without access to query labels?"

Even after reading the intro, I didn't fully grasp what this statement means, or why it is a natural question to ask. In general, there was little intuition for why this should work or what is going on. As I understand it, the representations are extracted from a frozen TFM where some examples (X, Y) are presented in context, and then X is presented again as a query. It is noted that "each query can thus access its own label through the context", but it's unclear to me why extracting "representations at the frozen decoder's input, where they depend on both the query and it's labeled block" should give us anything useful. Moreover, it is unclear to me (in Figure 1) why we should expect to be able to "denoise" an existing representation (which uses different in-context examples) back to this representation.

To the second point, this method (training the ~16M parameter denoiser) requires substantial additional training beyond using an off-the-shelf TFM, but doesn't show much benefit compared to existing TFMs like TabPFN-v3. There are extremely small differences (four significant digits!) in Table 1 compared to TabPFN-v3, which seems to perform best, while the proposed method is also second-best on F1-macro and Acc. rank. It seems like there is a slightly lower runtime vs TabPFN-v3 in Figure 2, but it is unclear if this is meaningful (about 1 second improvement on average?). Moreover, the proposed method loses against TabPFN-v3 more often than it wins.

Pairwise win rates. Figure 2(a) shows that TabSQRT wins on 62% of datasets against LimiX. Against TabPFN-v3, TabSQRT wins on 43% and TabPFN-v3 on 57%, despite their equal mean accuracy.

It also seems (from Table 2) that FT-LimiX, a fine-tune of LimiX (the backbone used here, with further fine-tuning presumably done on the same training data as the proposed approach?) performs about as well (accuracy of 88.29 vs. 88.65) as the proposed method, which further makes me question the value of the specific approach taken here.

Questions for authors

Can the authors give me some intuition for why they would expect their procedure to work? What is the signal that is hopefully being learned here by the de-noiser model?

Reviewer_YpnYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, authors propose a new way of training a Tabular Foundation Model (TFM), using self query representations as a source of supervision. To obtain a set of self-query representations for a set of objects (x_i, y_i), authors provide them both as a context (including y_i), and as a query (without y_i) to a frozen LimiX model. They then collect the query representations by passing the query subset through the model. After that, the authors train a conditional diffusion student model, which learns to generate the query representation conditioned on standard context features and labels, as well as the features of a query. As a post-training procedure, authors finetune their model with adding a standard cross-entropy target. The authors finetune a model on a set of 100000 real-life datasets. They then demonstrate superior performance across 4 metrics on TabArena.

Critical strengths
  1. The paper uses a surprising source of useful supervision. The idea that passing the same set of data as both the query and the context provides a useful signal that could be utilized for model improvement is surprising and novel. Additionally, the fact that this procedure leads to the improved performance could hint at a minor flaw in the development of LimiX.
  2. The evaluation is extensive, with the model performance evaluated on TabArena and CC18. Ablation studies are also extensive, demonstrating usefulness of both conditional denoising and cross-entropy losses.
Critical weaknesses
  1. The ablations demonstrate that each part of the existing pipeline is useful, but they do not show that simpler and more effective alternatives don't exist. Specifically, it is unclear if conditional diffusion model is optimal, or whether similar or even larger improvements could be obtained through simpler objectives.
Questions for authors
  1. Have you tried any simpler baselines, that use alternative distillation techniques instead of the diffusion model?
  2. Since the performance difference is small, have you compared against different hyperparameter configurations of LimiX finetuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.