acceptodds
Under review as a conference paper at ICLR 2027

When Does Training Beat Prompting on Tabular Data?

Abstract

Choosing between a prompted language model, a tabular foundation model, and a trained classical model requires understanding where their learning curves exchange advantage and how reproducible that boundary is. We study label-budget crossovers on 18 tabular datasets using 3 splits, shared evaluation rows, and training budgets from 8 to 2,048 labels. Equal-label comparisons are distinguished from comparisons against a fixed 32-shot LLM. We report observed crossover brackets, outcomes outside the measured range, and conditional stability under paired resampling of evaluation rows. For CatBoost classification against GPT-4.1-mini, median bootstrap retention is 57.8% for the exact bracket or censored endpoint, but 99.45% for the broad crossover category: uncertainty in location need not imply uncertainty in whether a crossing is observed. Model choice also changes qualitative outcomes. On the same split, CatBoost achieves lower error than 32-shot GPT-4.1-mini at some measured budget on all 8 regression datasets, but does not overtake 32-shot GPT-5.5 within the evaluated range on 5. A separate retrospective analysis of 126 implementations based on a shared starter protocol examines variation across implementations. These findings motivate reporting crossover location, outcome stability, and model dependence separately when assessing label efficiency in tabular prediction.

Primary area:
datasets and benchmarks
Keywords:
tabular prediction, label efficiency, learning curves, large language models, tabular foundation models, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U27POpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary
  • This paper studies label-budget crossovers between prompted LLMs and trained models, e.g., CatBoost, on 18 tabular benchmarks, with all methods scored on the same evaluation rows.
  • It finds that a crossover occurs highly stably but its location is highly uncertain.
  • It also analyzes variation across a separate cohort of 126 implementations built from identical starter code.
Critical strengths
  • The paper uses a fair experimental design by fixing the evaluation rows before any model is run, so that all methods are compared on the same rows.
  • It draws conclusions separately about the crossover occurrence and its location, making the findings clear.
  • The paper covers both regression and classification tasks and conducts sensitivity analyses on the number of shots.
Critical weaknesses
  • The contribution is limited. Similar comparisons between LLMs and trained tabular models across label budgets have already been conducted, e.g., TabLLM and TabuLa-8B. Also, the paper does not analyze the factors behind earlier or later crossovers (e.g., number of features, memorization), so the “when” in the title is only partially addressed.
  • All evaluated LLMs are OpenAI models, and only gpt 4.1 mini is evaluated on all three splits. Other API or open-weight models would make the conclusions more general.
  • The prompting setup is not explored sufficiently. Class balancing is not considered, and additional shots barely help in classification (details in Questions), which suggests the prompt may be suboptimal and the crossover may be biased toward smaller label budgets.
  • CatBoost uses up to 2k labels while the LLM is fixed at 32 shots, so it is unclear whether the crossover reflects a limitation of LLMs or of the 32-shot design.
Questions for authors
  • How much do the crossover results change with different few-shot samples? With label-blind sampling, even with 32 shots, imbalanced datasets may include very few minority class samples. In regression, do the examples fairly cover the target range? Results with class-balanced/target-stratified examples would be helpful.
  • Do the crossover occurrence and location change with the prompt design? Additional shots barely help in classification, which differs from prior findings. How much of the prompt design space was explored before the prompt was frozen?
Reviewer_coygOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the learning curve crossover between LLMs with prompt (GPT-4/5), tabular foundation models, and classical models across 18 datasets with label budgets from 8 to 2048. The authors separate broad crossover outcome categories from exact bracket locations under paired evaluation-row bootstrap resampling, and complement this benchmark with a retrospective analysis of 126 implementations produced under a shared starter protocol. They conclude that there is no single universal label threshold where training beats prompting: while whether a crossover occurs is mostly stable to test-row resampling, the exact crossover budget is noisy and qualitative outcomes shift substantially depending on the specific LLM (e.g., gpt-4.1-mini vs. gpt-5.5) and implementation pipeline choices.

Critical strengths
  1. Separating broad crossover outcomes (99.45% bootstrap retention) from exact label-budget locations (57.8% retention) is an interesting and practical observation for benchmarking.
  2. Analyzing implementation variance across 126 shared-starter submissions provides a helpful perspective on reproducibility in tabular ML benchmarks.
Critical weaknesses
  1. Compared to standard tabular benchmarks, 18 datasets is relatively small, and 8 of them exhaust their training pool before the 2048-label budget.
  2. Hard to follow in places due to execution issues and single-split frontier runs: The presentation is dense and somewhat hard to follow, and several experimental issues
Questions for authors
  1. How do the low-budget (N=8) comparisons and crossover brackets change if stratified sampling is used so that every training subset contains both classes?
  2. Do the gpt-5.5 crossover results in Table 2 hold when evaluated across all three splits?
Reviewer_vyomOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when trained tabular models outperform prompted LLMs as the number of labeled examples increases. The authors compare classical models, TabPFN, and prompted LLMs on 18 tabular datasets. They study not only where the performance curves cross, but also how stable the crossover outcomes and locations are under bootstrap resampling. The paper further examines model dependence and analyzes variation across 126 implementations collected under a separate historical protocol.

The paper studies a meaningful practical question and presents a reasonably careful empirical analysis. However, I am not fully convinced that the reported crossover stability generalizes beyond the specific sampling strategies, models, and datasets considered. Further analysis of training-subset selection and cross-model variability would strengthen the conclusions.

Critical strengths
  1. The paper addresses a practically relevant problem. Understanding when training becomes more effective than prompting could help practitioners make better decisions about model selection, potentially reducing human annotation and training costs.
  2. The experimental design is relatively careful. The use of shared evaluation rows, matched labeled examples at small budgets, multiple data splits, and paired bootstrap resampling makes the comparisons more controlled and provides useful evidence about crossover stability.
  3. The authors provide a detailed discussion of experimental limitations and potential sources of uncertainty. In particular, they acknowledge the dependence on model choice, differences between the controlled and historical protocols, and several execution issues. This transparency helps clarify the scope of the findings.
Critical weaknesses
  1. The crossover analysis focuses on label quantity but does not sufficiently consider the selection of labeled examples. The training subsets are constructed through random permutations, while the informativeness and diversity of selected examples may substantially affect model performance. A small, carefully selected subset could potentially outperform a much larger random subset. The observed crossover locations may depend on the sampling strategy rather than only on the number of labels. Moreover, the bootstrap analysis resamples evaluation rows but does not capture variability from different training-subset selections. It would be useful to examine crossover stability under repeated subset sampling or alternative selection strategies.
  2. The generalizability of the crossover stability findings across models and datasets remains insufficiently explored. The authors acknowledge model dependence and provide comparisons across different model versions. However, the prompted LLMs are all from the same provider, and the stronger-model comparisons are limited to a single split. Different model families may exhibit different sensitivities to particular datasets. In addition, the use of public datasets raises the possibility of different levels of pretraining exposure, which could influence the observed performance and crossover behavior. Although the authors discuss this limitation, broader comparisons across independently developed model families and datasets with different characteristics would help establish how general the stability findings are.
Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.