acceptodds
Under review as a conference paper at ICLR 2027

Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens

Abstract

Every tabular classification task requires sufficient labeled data. When labels are scarce, practitioners can collect more examples, encode domain expertise into features, or accept lower predictive performance. Large language models offer a potential alternative by leveraging knowledge acquired during pretraining. This possibility raises three distinct questions: (i) can zero-shot inference reduce the need for labeled data; (ii) can a richer prompt reduce the need for collecting more examples; (iii) and can encoded domain knowledge compensate for missing data? To answer each question under explicit, testable conditions, we introduce TabLens, a benchmark of 91 real-world and synthetic datasets across eight domains, together with eight industrial datasets containing free-text columns. First, below a complexity threshold that can be estimated from a dataset's intrinsic dimensionality before running an LLM, zero-shot inference matches the accuracy that would otherwise require up to approximately 16 labeled examples to collect. This holds whether the model generates a written answer or, more cheaply, scores candidate labels directly from their log-probabilities – the latter loses almost no accuracy except for the smallest models. Above the threshold, this advantage disappears. However, on datasets with free-text columns, zero-shot LLMs are even better than 16-shot TabPFN-3 Plus, and no longer require manual feature engineering. Second, a richer prompt reduces the number of required demonstrations only when it includes both the real column names and a plain-language task description. Providing only one of these sources of context can perform worse than providing neither, particularly at higher shot counts. Third, on the LLM-synthetic datasets, explicit if–then expert decision rules improve performance only when the prompt provides sufficient task context for the model to interpret them. Even then, they only substitute for up to about 16 real examples, after which collecting more data is more effective. Together, these results establish testable conditions under which LLMs can reduce data-collection and feature-engineering requirements for tabular classification.

Primary area:
datasets and benchmarks
Keywords:
ICL, LLM, Zero-shot, Few-shot, Tabular Data, Binary Classification

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_vCNMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SummaryThis paper introduces TabLens, a benchmark designed to investigate whether LLMs can leverage their pretrained knowledge to reduce the need for labeled data in tabular classification tasks. TabLens consists of 99 datasets organized into five groups:

  1. Real-world tabular: Existing datasets collected from OpenML and other sources.
  2. Post-cutoff tabular: Datasets released after October 2024 to reduce the risk of pretraining contamination.
  3. LLM-generated synthetic: Synthetic datasets generated by LLMs with explicit decision rules.
  4. MLP-generated synthetic: Synthetic datasets generated using randomly initialized MLPs with varying complexity.
  5. Text-tabular: Eight datasets containing both structured features and free-form text. The authors investigate three main research questions:
  6. Can zero-shot inference substitute for collecting labeled data?
  7. Can richer prompt context substitute for collecting additional examples?
  8. Can encoded expert knowledge substitute for a lack of data? The authors find that zero-shot LLMs can achieve performance comparable to TabPFN using approximately 16 labeled examples on certain datasets. They also find that richer prompts, particularly those containing meaningful column names and task descriptions, can substantially improve zero-shot performance, although these benefits diminish as more labeled examples become available. Finally, they show that explicit expert knowledge can improve performance on LLM-generated synthetic datasets when combined with sufficient task context.
Critical strengths
  1. The research questions are interesting and practically relevant. Understanding whether pretrained LLM knowledge can compensate for limited labeled data is an important question for tabular classification, particularly in applications where collecting labels is expensive.
  2. The introduction of TabLens is a useful contribution to the research community. The benchmark brings together diverse real-world and synthetic datasets and provides a systematic evaluation of LLMs under different data availability, prompt context, and dataset complexity settings.
  3. The controlled experiments on prompt context provide useful empirical insights. In particular, the authors investigate the individual and combined effects of semantic column names, task descriptions, and few-shot demonstrations. The finding that richer context provides diminishing benefits as the number of demonstrations increases is interesting.
Critical weaknesses
  1. The ROC-AUC evaluation protocol is problematic and may lead to inflated results. In Section 3 (line 180), the authors replace ROC-AUC values below 0.5 with 1−AUC1-\mathrm{AUC}. This transformation is problematic because it systematically inflates reported AUC values. Even a random classifier would have an expected reported AUC greater than 0.5 after this transformation. Furthermore, determining whether the predictions should be inverted requires access to the ground-truth labels, which would not be available in a realistic zero-shot classification setting. This also raises concerns about the statistical significance tests comparing zero-shot performance against random guessing. If the transformed AUC values are used, testing whether the reported AUC is significantly greater than 0.5 may not be statistically valid.
  2. The experimental setup in Section 6 does not convincingly support the claim that expert knowledge can substitute for labeled data. The authors evaluate expert knowledge using LLM-generated synthetic datasets, where the ground-truth labels are determined by three explicit decision rules. However, the expert-knowledge prompts appear to provide these same rules, or closely related rules, to the LLM. If the provided rules are identical to those used to generate the ground-truth labels, the classification task becomes trivial, as a deterministic implementation of the rules should achieve perfect accuracy. More importantly, this setup is not representative of real-world classification tasks, where expert knowledge typically consists of approximate heuristics rather than the exact ground-truth labeling function. Therefore, the results do not convincingly demonstrate that expert knowledge can replace labeled examples in realistic classification settings.
  3. The text-tabular experiments lack important details about how text features are processed for tabular foundation models. In Section 4.3, the authors report that zero-shot LLMs substantially outperform few-shot TabPFN on text-only datasets. However, the paper does not clearly explain how free-text columns are processed and provided to TabPFN. This is important because TabPFN is primarily designed for structured tabular data. If raw text is simply treated as categorical features without preserving semantic information, its poor performance on text-only datasets would not be surprising. A more meaningful comparison would include a baseline that first extracts semantic text embeddings using a pretrained language model and then uses TabPFN or another tabular classifier for prediction. Without such comparisons, it is difficult to determine whether the observed advantage comes from the LLM's zero-shot classification capabilities or simply from the lack of appropriate text representations in the tabular baselines.
Questions for authors
  1. ROC-AUC evaluation: Why do the authors replace ROC-AUC values below 0.5 with 1−AUC1-\mathrm{AUC}? Could the authors report the original ROC-AUC values and clarify whether this transformation was also applied before conducting the statistical significance tests?

  2. Expert knowledge: In Section 6, are the expert rules provided to the LLM identical to the rules used to generate the ground-truth labels, or are they independently generated approximations? If they are identical, why does the model not achieve perfect classification performance? Could the authors additionally evaluate the effectiveness of expert knowledge on real-world datasets, where the ground-truth labels are not generated from explicit rules?

  3. Text-tabular classification: How are free-text features processed and fed into TabPFN and the other tabular baselines? Could the authors include experiments using pretrained text embeddings as input features to ensure a more meaningful comparison?

Reviewer_ysDmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TabLens, a benchmark of 99 binary tabular datasets (24 real-world, 8 post-cutoff, 24 LLM-synthetic, 35 MLP-prior syntheic, and 8 text-tabular), and evaluates tabular baselines (tree-based models and TFMs) and 5 LLMs (Qwen3, GPT-4o, GLM-4.7) in various zero/few-shot settings. Under this framework, the paper investigates zero-shot inference, rich prompt context, and explicit if-then decision rules can substitute labelled data. The paper concludes that 1. zero-shot LLMs can substitute labelled data when the dataset's intrinsic dimension is below a certain threshold, 2. only column names and task description together can substitute a need for more data, and 3. expert rules only help when accompanied with prior context.

Critical strengths
  1. The paper investigates in-context tabular ML from multiple perspectives: dataset complexity (via model-free intrinsic complexity), task context (table schema vs. task description), and encoded knowledge (prior context and expert knowledge).
  2. The use of post-cutoff datasets addresses practical deployment questions.
Critical weaknesses
  1. Post-hoc inversion of AUC (replace with 1.0-AUC when AUC < 0.5) per-model makes less sense then inverting AUCs of all models being compared together when we comparing models. Do the conclusions change with the un-flipped ROC-AUC?
  2. Manuscript inconsistency
  • In Section 4.1, reported performance is not found in Table 12, which is a duplicated verbatim of Table 3.
  • In section 6.1, Table 21 is a duplicate of Table 5. Then why referring Table 21 instead of 5?
  • Appendix ?? appears in line 168.
Questions for authors

See the weakness section

Reviewer_dFgtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the scarcity of labeled data for tabular classification. When labels are scarce, practitioners can collect additional labeled examples; the paper explores using large language models as an alternative to manual annotation. It presents a series of experiments examining practical aspects of LLM-based labeling, including zero-shot classification capabilities and the impact of the amount of information provided in prompts on performance.

Critical strengths

The paper investigates when LLMs can be used to predict labels for tabular data and systematically examines how task descriptions, few-shot labeled examples, and explicit expert rules contribute to performance.

Critical weaknesses
  1. The paper replaces test-set AUC values below 0.5 with 1-AUC, effectively using the test labels to select the prediction direction post hoc. This converts below-chance random fluctuations into above-chance scores, introducing an upward bias in the reported performance. If the subsequent Wilcoxon tests use these transformed AUC values, the claim of statistically significant performance above chance is invalid.
  2. In Table 17, the stronger zero-shot LLMs achieve mean AUC values closer to that of 8-shot TabPFN (0.721) than to that of 16-shot TabPFN (0.772). The claim that zero-shot LLMs approach the performance of 16-shot TabPFN-3 Plus is not consistently supported.
  3. The results described in the main text for the full 32-dataset pool are inconsistent with Table 12. In particular, the text reports an AUC of 0.753 for 16-shot TabPFN-3 Plus, whereas the table reports 0.732.
  4. Line 168 contains an unresolved reference, “Appendix ??”. Tables 3–5 and Figure 1 are not explicitly referenced in the main text. Table 12 duplicates Table 3; the corresponding main-text reference should therefore point to Table 3, and the redundant table should be reconsidered. The manuscript also contains numerous language issues, such as the unclear expression “with AUC 1-AUC”, and requires thorough proofreading.
  5. The novelty is limited. Using LLMs for few-shot and zero-shot tabular classification has already been explored. The paper’s main contribution should therefore be to establish when LLMs can reduce labeling requirements and what task information they need. However, the current evaluation methodology and experimental results do not adequately support these conclusions.
Questions for authors

See Weaknesses.

Reviewer_mbg5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TabLens, a benchmark with 99 binary tabular classification datasets for studying when LLM knowledge can substitute for labeled data. The paper is organized around three questions: whether zero-shot LLMs can replace collecting labeled data, whether richer prompt context can replace demonstrations, and whether encoded expert rules can replace data. Five LLMs are compared against TabPFN-3 Plus and other tabular baselines using ROC-AUC.

The main conclusions are that zero-shot LLMs roughly match 16-shot TabPFN below an "intrinsic-dimensionality" threshold (and do especially well when an important signal is in text based entries/columns), that prompt context helps only when both column names and a task description are present, and that expert rules help only when paired with a task description and only substitute for about 16 examples worth of data.

Critical strengths

The paper manages to draw systematic conclusions about LLM performance on tabular data, which is valuable. In my experience it's difficult to find anything systematic about model performance across disparate collections of tabular datasets, and framing the results as conditions under which an LLM lets a practitioner skip a specific cost (data, demonstrations, or expert knowledge) makes them more actionable than an aggregate leaderboard.

The text-tabular result is also convincing and practical: zero-shot LLMs clearly beat TabPFN and CatBoost when the predictive signal sits in free-text columns, but LLM knowledge is not a good substitute for more data once datasets have a few dozen samples.

Critical weaknesses
  1. Many of the conclusions are based on small performance differences, and the results are not always consistent across tables. For example, the text and Table 3 report different averages on the 32-dataset pool (TabPFN 0.753 vs 0.732, GPT-4o-mini 0.712 vs 0.722, Qwen3-14B 0.714 vs 0.693), the claim that incomplete context is worse than none rests on a gap of about 0.01 to 0.02 AUC in one panel, and the real-data complexity trend has a slope of about -0.01 with a lot of scatter. Im not convinced these results are statistically significant, and even if they are the effect size is small enough that one should probably not rely upon them when analyzing any one specific dataset.

One statistical issue that might matter (I can't tell how big of an effect this had): Any ROC-AUC below 0.5 is replaced with 1 - AUC (lines 177-180). A zero-shot model has no way to know its predictions should be flipped without labels, so this inflates zero-shot scores, most of all near chance, and it means the Wilcoxon test against random guessing in H1a I guaranteed to reject the null hypothesis with a large enough sample size.

These issues make it hard to trust the specific thresholds the paper is built around, such as zero-shot being worth about 16 labeled examples. Figure 3 suggests it is closer to 4 to 8 for TabPFN, unless I'm misinterpreting something?

  1. The expert-knowledge experiment (Q3) is circular. The synthetic datasets are generated by an LLM from decision rules, and those same rules are then given to the model as expert knowledge. That is essentially handing the model the data-generating process, so the result that expert rules plus a task description reach 0.86 AUC says little about whether real domain expertise can substitute for data. TBH I can't think of a better way to do this experiment since it must be in a domain where logical rules are known and can be provided. But at the same time I think the conclusions of the experiment are more or less foretold by the way it is constructed.

Many of the main findings also seem like reconfirmations of results already in the literature rather than new discoveries. TabLLM (Hegselmann et al., 2023) showed examples where zero-shot LLMs can be competitive with few-shot tabular learners when columns have meaningful names. TABLET (Slack & Singh, 2023) showed that natural-language task descriptions help. The benefit of LLMs on tables with free-text columns is a central motivation of prior text-tabular benchmarks (Shi et al., 2021; Mraz et al., 2025). The substitution framing is a nice way to organize these effects, but it would help if the paper were clearer about how these conclusions relate to, reinforce, or contradict prior work.

A minor point: there are broken cross-references that show up as "??" (e.g. line 168), including the one that should explain why GLM has only partial coverage of the benchmark. This is NBD and doesn't impact my assessment but worth fixing when you revise.

Questions for authors

Clarifications: How many dataset-level AUCs were flipped, and what do the main results and the H1a test look like without flipping? Why do the numbers in the text and Table 3 disagree, and which of the main takeaways survive per-dataset significance testing?

Suggestion: The Q3 experiment would be much more convincing with rules written independently of the data generator, for example real expert rules on real datasets. Or maybe generating useful logical rules on real data using a simple decision tree. I'm not sure how these suggestions would pan out though - Its not entirely clear to me how to do Q3 in a scientific way with real data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.