TabSQRT: Self-Query Representation Transfer for Tabular In-Context Learning

Tabular foundation models (TFMs) predict query labels from labeled context examples; during training, the query labels themselves serve only as loss targets. We ask a different question: if a frozen TFM is allowed to see the answers, can it teach a student how to represent examples whose answers are hidden? TabSQRT (Self-Query Representation Transfer) turns this idea into supervision. TabSQRT constructs targets by supplying each labeled training block to a frozen TFM as both context and queries. A conditional diffusion student learns to generate the resulting decoder-input representations from standard in-context inputs, followed by classification post-training through the frozen decoder. In diagnostics with students trained separately on each dataset, larger target-construction blocks improve diffusion-student accuracy even as target decoding accuracy declines. This finding motivates assessing supervision through downstream student performance. Trained across 100,000 real-world tables, TabSQRT achieves 88.65% mean accuracy on 38 TabArena classification datasets without dataset-specific fine-tuning, compared with its LimiX teacher's 88.30%, and ties the highest reported mean accuracy among current frontier TFMs. Ablations support combining self-query targets, diffusion-based representation learning, and classification post-training for transfer to unseen tables.