- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SummaryThis paper introduces TabLens, a benchmark designed to investigate whether LLMs can leverage their pretrained knowledge to reduce the need for labeled data in tabular classification tasks. TabLens consists of 99 datasets organized into five groups:
- Real-world tabular: Existing datasets collected from OpenML and other sources.
- Post-cutoff tabular: Datasets released after October 2024 to reduce the risk of pretraining contamination.
- LLM-generated synthetic: Synthetic datasets generated by LLMs with explicit decision rules.
- MLP-generated synthetic: Synthetic datasets generated using randomly initialized MLPs with varying complexity.
- Text-tabular: Eight datasets containing both structured features and free-form text. The authors investigate three main research questions:
- Can zero-shot inference substitute for collecting labeled data?
- Can richer prompt context substitute for collecting additional examples?
- Can encoded expert knowledge substitute for a lack of data? The authors find that zero-shot LLMs can achieve performance comparable to TabPFN using approximately 16 labeled examples on certain datasets. They also find that richer prompts, particularly those containing meaningful column names and task descriptions, can substantially improve zero-shot performance, although these benefits diminish as more labeled examples become available. Finally, they show that explicit expert knowledge can improve performance on LLM-generated synthetic datasets when combined with sufficient task context.
Critical strengths
- The research questions are interesting and practically relevant. Understanding whether pretrained LLM knowledge can compensate for limited labeled data is an important question for tabular classification, particularly in applications where collecting labels is expensive.
- The introduction of TabLens is a useful contribution to the research community. The benchmark brings together diverse real-world and synthetic datasets and provides a systematic evaluation of LLMs under different data availability, prompt context, and dataset complexity settings.
- The controlled experiments on prompt context provide useful empirical insights. In particular, the authors investigate the individual and combined effects of semantic column names, task descriptions, and few-shot demonstrations. The finding that richer context provides diminishing benefits as the number of demonstrations increases is interesting.
Critical weaknesses
- The ROC-AUC evaluation protocol is problematic and may lead to inflated results. In Section 3 (line 180), the authors replace ROC-AUC values below 0.5 with . This transformation is problematic because it systematically inflates reported AUC values. Even a random classifier would have an expected reported AUC greater than 0.5 after this transformation. Furthermore, determining whether the predictions should be inverted requires access to the ground-truth labels, which would not be available in a realistic zero-shot classification setting. This also raises concerns about the statistical significance tests comparing zero-shot performance against random guessing. If the transformed AUC values are used, testing whether the reported AUC is significantly greater than 0.5 may not be statistically valid.
- The experimental setup in Section 6 does not convincingly support the claim that expert knowledge can substitute for labeled data. The authors evaluate expert knowledge using LLM-generated synthetic datasets, where the ground-truth labels are determined by three explicit decision rules. However, the expert-knowledge prompts appear to provide these same rules, or closely related rules, to the LLM. If the provided rules are identical to those used to generate the ground-truth labels, the classification task becomes trivial, as a deterministic implementation of the rules should achieve perfect accuracy. More importantly, this setup is not representative of real-world classification tasks, where expert knowledge typically consists of approximate heuristics rather than the exact ground-truth labeling function. Therefore, the results do not convincingly demonstrate that expert knowledge can replace labeled examples in realistic classification settings.
- The text-tabular experiments lack important details about how text features are processed for tabular foundation models. In Section 4.3, the authors report that zero-shot LLMs substantially outperform few-shot TabPFN on text-only datasets. However, the paper does not clearly explain how free-text columns are processed and provided to TabPFN. This is important because TabPFN is primarily designed for structured tabular data. If raw text is simply treated as categorical features without preserving semantic information, its poor performance on text-only datasets would not be surprising. A more meaningful comparison would include a baseline that first extracts semantic text embeddings using a pretrained language model and then uses TabPFN or another tabular classifier for prediction. Without such comparisons, it is difficult to determine whether the observed advantage comes from the LLM's zero-shot classification capabilities or simply from the lack of appropriate text representations in the tabular baselines.
Questions for authors
-
ROC-AUC evaluation: Why do the authors replace ROC-AUC values below 0.5 with ? Could the authors report the original ROC-AUC values and clarify whether this transformation was also applied before conducting the statistical significance tests?
-
Expert knowledge: In Section 6, are the expert rules provided to the LLM identical to the rules used to generate the ground-truth labels, or are they independently generated approximations? If they are identical, why does the model not achieve perfect classification performance? Could the authors additionally evaluate the effectiveness of expert knowledge on real-world datasets, where the ground-truth labels are not generated from explicit rules?
-
Text-tabular classification: How are free-text features processed and fed into TabPFN and the other tabular baselines? Could the authors include experiments using pretrained text embeddings as input features to ensure a more meaningful comparison?