TabMoC: Scaling Tabular Foundation Models to Large Datasets via Mixture-of-Contexts

Tabular foundation models (TFMs) have emerged as strong predictors for diverse tabular tasks, yet their inference is typically limited to relatively small datasets. This limitation arises because TFMs condition on training data at inference time, making full-context inference increasingly memory-intensive as the training set grows. Existing approaches alleviate this cost by selecting only a small subset of the training data as context, which leave much of the available training information unused and therefore limit the predictive performance of TFMs. To address this issue, we investigate large-scale tabular prediction with TFMs through Mixture-of-Contexts (TabMoC). TabMoC partitions the training set into multiple small chunks and processes them separately with a tabular foundation model. A learned router then integrates these contexts across chunks to produce the final prediction. In this way, TabMoC respects the context-size constraint while drawing on information beyond a single context. Experiments demonstrate that TabMoC more closely approaches the performance of full-context inference, indicating that it can better exploit the potential of TFMs on large datasets. Moreover, as a compatible method, TabMoC consistently improves performance across multiple TFMs on both synthetic and real-world datasets.