acceptodds
Under review as a conference paper at ICLR 2027

TabMoC: Scaling Tabular Foundation Models to Large Datasets via Mixture-of-Contexts

Abstract

Tabular foundation models (TFMs) have emerged as strong predictors for diverse tabular tasks, yet their inference is typically limited to relatively small datasets. This limitation arises because TFMs condition on training data at inference time, making full-context inference increasingly memory-intensive as the training set grows. Existing approaches alleviate this cost by selecting only a small subset of the training data as context, which leave much of the available training information unused and therefore limit the predictive performance of TFMs. To address this issue, we investigate large-scale tabular prediction with TFMs through Mixture-of-Contexts (TabMoC). TabMoC partitions the training set into multiple small chunks and processes them separately with a tabular foundation model. A learned router then integrates these contexts across chunks to produce the final prediction. In this way, TabMoC respects the context-size constraint while drawing on information beyond a single context. Experiments demonstrate that TabMoC more closely approaches the performance of full-context inference, indicating that it can better exploit the potential of TFMs on large datasets. Moreover, as a compatible method, TabMoC consistently improves performance across multiple TFMs on both synthetic and real-world datasets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Tabular Data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X13QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a mixture-of-context approach to solve the problem of inferencing tabular PFNs on large datasets. First, they split the dataset into mutually-exclusive subsets via K-Means. Each test query is predicted separately based on each context. The predictions are then aggregated with weights predicted by a separate router network. The router network is trained on synthetic data. On 130 real datasets, the proposed method appears ahead of the alternative context compression method (single-context retrieval and uniform averaging) but behind the full context.

Critical strengths
  1. The paper addresses an important problem of long-context in the modern application of tabular PFN models. Using the proposed method, we can reduce the computation cost from O(N2)O(N^2) to O(C∗Nc2)O(C * N_c^2), where NcN_c is the chosen context limit.
  2. The proposed architecture for the router network and its loss function appears quite novel. First, the authors identify a set of descriptor statistics for each subset based on which they route the test sample, e.g., they use (i) distance to the cluster and the nearest neighbor; (ii) for classification, they add predictive confidence, agreement with prior mixture, and class distribution; (iii) for regression, they use various deviations from the prior. Second, they design a loss that besides the predictive power also optimizes (i) advantage over the geometric prior; (ii) penalizes the deviations from the geometric prior.
  3. Empirically, on a set of 130 datasets, the proposed method seems to marginally outperform uniform averaging.
Critical weaknesses
  1. Datasets size. Largest included dataset (regression) has 32,142 labeled rows, while largest classification dataset is even smaller and has 16,000 train samples. Current generation of PFN models (e.g. TabPFN v3.5 or v3, Causilo, Kumo, Limix-2) can handle such context size out-of-the-box. So, it's hard to draw conclusions about the usefulness of the method based on the provided results as they don't cover the datasets where the context length becomes problematic.
  2. Training set scaling. From Figure 3, scaling curve for the proposed method vs. full-context appears to be flatter, i.e., the proposed method seems to scale poorly with the size of the training set. This concern together with the above again undermines the usefulness of the method in practice further.
  3. Geometric prior is good enough, it does not seem like training a prior justifies its cost. From Tables 7-8, the performance boost from using the router vs. the geometric prior appears to be roughly 0.3%. It's not convincing that using the router is strictly beneficial, especially given the finite compute budget.
  4. Statistical significance of the results. Since some of the reported numbers are quite close together, it's especially important to understand stochasticity in the results. To this end, the authors don't disclose whether they have re-run the models with different seeds, (how many times?), used cross-validation, or re-run with different splits. There are no standard deviation / errors or CI's provided.
  5. Metrics. The authors report average RMSE and AUC across regression and classification datasets respectively. I am not entirely convinced of the choice of metric. Of course, all aggregate metrics tend to conceal something. So, I think it's important to provide at least a few or report per-dataset errors in the supplement. In the current literature on tabular learning (TabArena, BeyondArena), the community has converged to Elo score, normalized error, winrate, improvability, relative improvement over XGBoost/MLP as the standard aggregate metric baselines.
  6. Computational cost. It's not clear how the method scales. The authors have replaced one attention computation with multi-step procedure: (i) run k-means (ii) run router (iii) run inference on each context separately (iv) compute the weighted average. In particular, depending on whether or not we can parallelize step (iii) and to what extent, the computational speed of the proposed method can differ drastically in practice. The multi-step cascade procedure is also harder to optimize on the GPU kernel level. It's important to provide both formal computational cost analysis as well as a comprehensive benchmark (e.g. what's the inference speed for batch size of 1?).
Questions for authors
  1. Can the authors provide standard deviations (CI's)? If not, can they share per-dataset errors? It would also be nice to see alternative aggregate metrics, e.g., Elo, winrate, normalized score, relative improvement.
  2. Can the authors provide analysis of the computational cost? It would also help a lot if they can provide empirical timed runs of their method vs. full-context, e.g., a plot of inference speed with fixed batch size vs. the number of training samples. How can we parallelize inference over multiple contexts?
Reviewer_nsDyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose to build a mixture of contexts approach for tabular foundation models, by building clusters of context points, which are combined from a router based on the query point of interest. The authors consider synthetic tasks, real classification, and regression tasks. The authors consider a few tabular foundation model architectures and advocate that the proposed approach achieves results close to using the full context and outperforms other baseline approaches in building a smaller context for inference, such as uniform sampling, nearest neighbor sampling, and the uniform mixture approach.

Critical strengths
  • The paper is well written and clear.
  • The authors consider several tabular foundation model architectures.
  • The authors consider regression and classification datasets.
Critical weaknesses
  • The authors have missed recent and relevant related work [1]
  • The authors do not use the most recent benchmarks established in the domain, TabArena [2] and Beyond [3].
  • While the authors mention that the current tabular foundation models have limitations on large dataset sizes, the exact impact of the proposed approach in the large dataset regime is not investigated. How would the proposed approach perform for very large datasets, compared to XGBoost/Catboost? Or to deep neural networks?
  • Aggregating via mean for section 4.2 and section 4.3 is one way, however, it might not show the real picture. Could the authors provide ranks for the different methods, otherwise a very good performance on a few datasets with a large gap to other methods can provide misleading aggregated results.
  • It would be interesting to provide the results for higher mm sizes for the real datasets, for TabMoC and the other baselines.
  • Are the authors using an ensemble size of 1 for the full context results? What is the ensemble size for TabMoC? I would like to better understand the relation of TabMoC to the recent tabular foundation model architectures, since by default they use ensembling with transformed dataset views.

[1] Zabërgja, G., Kamel, R., Kadra, A., Frey, C. M., & Grabocka, J. (2026). End-to-end compression for tabular foundation models. Forty-third International Conference on Machine Learning

[2] Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Desai, P., Salinas, D., & Hutter, F. (2026). Tabarena: A living benchmark for machine learning on tabular data. Advances in Neural Information Processing Systems, 38.

[3] Purucker, L., Tschalzev, A., Erickson, N., Blayer, G., Holzmuller, D., Arazi, A., ... & Hutter, F. (2026). Beyond IID: How General Are Tabular Foundation Models, Really?. NeurIPS 2026

Questions for authors
  • How does the proposed approach perform compared to TACO [1] when the full context is used as input for TACO and the compressed latent dataset is equal to 200 for subsequent inference calls? How does it perform over varying mm sizes? What is the time performance?
  • How does the proposed approach perform in large dataset regimes [2][3] to XGBoost/MLP architectures?
  • How does the proposed approach compare to other compression baselines when ranks are generated?
  • How does the proposed approach compare in terms of time to other compression baselines?
  • How does the proposed approach perform when varying context sizes are used for real datasets compared to the other baselines?
  • What ensemble size is used for TabMoC and how is it related to the current ensembling approaches with transformed dataset views that current tabular foundation models use?

[1] Zabërgja, G., Kamel, R., Kadra, A., Frey, C. M., & Grabocka, J. (2026). End-to-end compression for tabular foundation models. Forty-third International Conference on Machine Learning

[2] Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Desai, P., Salinas, D., & Hutter, F. (2026). Tabarena: A living benchmark for machine learning on tabular data. Advances in Neural Information Processing Systems, 38.

[3] Purucker, L., Tschalzev, A., Erickson, N., Blayer, G., Holzmuller, D., Arazi, A., ... & Hutter, F. (2026). Beyond IID: How General Are Tabular Foundation Models, Really?. NeurIPS 2026

Reviewer_rf8gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TabMoC (Mixture-of-Contexts) to address context-length limitation in tabular foundation models (TFMs). It partitions a training dataset into disjoints chunks of size <= m (the maximum context of TFM) via k-means in the feature space, obtains predictions from TFM conditioned on each chunk, and aggregates the predictions using weights predicted by a pre-trained router. TabMoC is evaluated with LimiX-16M, TabPFN-2.5, and TabICLv2 on real and synthetic datasets (80 classification and 50 regression).

Critical strengths
  1. Mathematically well-behaved formularization: Framing clusters of the training dataset as context experts and router's log-weight adjustment (multiplying exp⁡δϕ,b(x)\exp \delta_{\phi,b}(\mathbf{x}) which is bounded by a factor of exp⁡2ρ\exp 2\rho), is simple and permutation-invariant.
  2. Utilizing synthetic prior in training the router: The router shares the same training framework as TFM, which allows it to adopt the future advances in TFM.
Critical weaknesses
  1. Marginal contribution of the learned router over parameter-free distance routing: The main text only compares with random context, nearest context (single chunk), uniform mixture (same weights). Appendix D (Table 7) reveals that comparing TabMoC agains parameter-free distance routing yields only a small improvement.
  2. Key baselines missing: Chunked TabPFN, Mixture of In-Context Prompters, and LoCalPFN are cited, but not evaluated as baseline.
Questions for authors
  1. Please see the weakness section.
  2. During training of the router and inference of TabMoC, the GPU memory can be bottlenecked by m and B. Does this still limit the maximum size of the training set applicable to the proposed method?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all the reviewers for their time, effort, and constructive feedback on our manuscript. We are also delighted that the reviewers recognize the significance of our research problem, the motivation behind our work, the method formulation, and the overall quality of the paper's presentation. At the same time, we note that the reviewers' primary concern centers on the lack of sufficient baseline comparisons. We would like to clarify that TabMoC is designed as a method that is compatible with all these approaches, without imposing any specific requirements on the inference strategies or model architectures. Nevertheless, we sincerely appreciate the reviewers' valuable suggestions, which will help us further strengthen and improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.