- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a mixture-of-context approach to solve the problem of inferencing tabular PFNs on large datasets. First, they split the dataset into mutually-exclusive subsets via K-Means. Each test query is predicted separately based on each context. The predictions are then aggregated with weights predicted by a separate router network. The router network is trained on synthetic data. On 130 real datasets, the proposed method appears ahead of the alternative context compression method (single-context retrieval and uniform averaging) but behind the full context.
Critical strengths
- The paper addresses an important problem of long-context in the modern application of tabular PFN models. Using the proposed method, we can reduce the computation cost from to , where is the chosen context limit.
- The proposed architecture for the router network and its loss function appears quite novel. First, the authors identify a set of descriptor statistics for each subset based on which they route the test sample, e.g., they use (i) distance to the cluster and the nearest neighbor; (ii) for classification, they add predictive confidence, agreement with prior mixture, and class distribution; (iii) for regression, they use various deviations from the prior. Second, they design a loss that besides the predictive power also optimizes (i) advantage over the geometric prior; (ii) penalizes the deviations from the geometric prior.
- Empirically, on a set of 130 datasets, the proposed method seems to marginally outperform uniform averaging.
Critical weaknesses
- Datasets size. Largest included dataset (regression) has 32,142 labeled rows, while largest classification dataset is even smaller and has 16,000 train samples. Current generation of PFN models (e.g. TabPFN v3.5 or v3, Causilo, Kumo, Limix-2) can handle such context size out-of-the-box. So, it's hard to draw conclusions about the usefulness of the method based on the provided results as they don't cover the datasets where the context length becomes problematic.
- Training set scaling. From Figure 3, scaling curve for the proposed method vs. full-context appears to be flatter, i.e., the proposed method seems to scale poorly with the size of the training set. This concern together with the above again undermines the usefulness of the method in practice further.
- Geometric prior is good enough, it does not seem like training a prior justifies its cost. From Tables 7-8, the performance boost from using the router vs. the geometric prior appears to be roughly 0.3%. It's not convincing that using the router is strictly beneficial, especially given the finite compute budget.
- Statistical significance of the results. Since some of the reported numbers are quite close together, it's especially important to understand stochasticity in the results. To this end, the authors don't disclose whether they have re-run the models with different seeds, (how many times?), used cross-validation, or re-run with different splits. There are no standard deviation / errors or CI's provided.
- Metrics. The authors report average RMSE and AUC across regression and classification datasets respectively. I am not entirely convinced of the choice of metric. Of course, all aggregate metrics tend to conceal something. So, I think it's important to provide at least a few or report per-dataset errors in the supplement. In the current literature on tabular learning (TabArena, BeyondArena), the community has converged to Elo score, normalized error, winrate, improvability, relative improvement over XGBoost/MLP as the standard aggregate metric baselines.
- Computational cost. It's not clear how the method scales. The authors have replaced one attention computation with multi-step procedure: (i) run k-means (ii) run router (iii) run inference on each context separately (iv) compute the weighted average. In particular, depending on whether or not we can parallelize step (iii) and to what extent, the computational speed of the proposed method can differ drastically in practice. The multi-step cascade procedure is also harder to optimize on the GPU kernel level. It's important to provide both formal computational cost analysis as well as a comprehensive benchmark (e.g. what's the inference speed for batch size of 1?).
Questions for authors
- Can the authors provide standard deviations (CI's)? If not, can they share per-dataset errors? It would also be nice to see alternative aggregate metrics, e.g., Elo, winrate, normalized score, relative improvement.
- Can the authors provide analysis of the computational cost? It would also help a lot if they can provide empirical timed runs of their method vs. full-context, e.g., a plot of inference speed with fixed batch size vs. the number of training samples. How can we parallelize inference over multiple contexts?
General Response
We sincerely thank all the reviewers for their time, effort, and constructive feedback on our manuscript. We are also delighted that the reviewers recognize the significance of our research problem, the motivation behind our work, the method formulation, and the overall quality of the paper's presentation. At the same time, we note that the reviewers' primary concern centers on the lack of sufficient baseline comparisons. We would like to clarify that TabMoC is designed as a method that is compatible with all these approaches, without imposing any specific requirements on the inference strategies or model architectures. Nevertheless, we sincerely appreciate the reviewers' valuable suggestions, which will help us further strengthen and improve our work.