acceptodds
Withdrawn from ICLR 2027

AutoTabBench: Automated Tabular Benchmark for Tabular Foundation Models with Evidence-Grounded Scope Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tabular foundation models (TFMs), pretrained on large-scale tabular data, have emerged as powerful data-analysis assistants for real-world applications, supporting data interpretation, prediction, and decision-making across diverse domains. As both general-purpose and domain-specific TFMs continue to grow, a natural question arises: how can we comprehensively and systematically evaluate their capabilities and determine which models are best suited to different tasks and conditions? Existing benchmarks provide the main basis for such comparisons, but they often reduce model evaluation to aggregate predictive performance over limited conditions. As a result, they provide limited evidence about how model advantages change across task types, distribution shifts, data characteristics, and perturbations. We argue that reliable evaluation requires making explicit the boundary of conditions under which a benchmark supports its conclusions. To this end, we introduce BenchScope, an evidence-grounded framework for benchmark scope analysis and design. BenchScope recovers verifiable evaluation evidence from papers, code, configurations, and manifests, and structures it across task, generalization, and protocol dimensions. Using explicit evidence requirements and decision rules, it characterizes benchmark boundaries by identifying covered conditions, missing coverage, protocol constraints, and unresolved evidence, and translates these findings into benchmark-construction requirements. Across six benchmarks, structured boundary recovery improves from 80.56% to 94.44%. Applying BenchScope to 37 benchmark entries, we construct AutoTabBench, an 80-task automated tabular benchmark covering routine prediction, natural distribution shifts, and controlled stress conditions, under which model advantages vary substantially across evaluation settings.

Primary area:
datasets and benchmarks
Keywords:
Tabular Foundation Models, Benchmark, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X2xnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that benchmarks for tabular foundation models rarely state the conditions under which their conclusions hold, such as which tasks, splits, and evaluation protocols they cover. It makes two contributions.

The first is BenchScope, a GPT-5.6 agent guided by a written “Boundary Skill” of 44 indicators. It reads a benchmark’s paper and code and produces a structured profile of what the benchmark tests, with each value linked to its source and marked as observed, inferred, or unknown. With the Skill, the agent’s accuracy on 20 profile fields rises from 80.56% to 94.44% across six benchmarks. The second is AutoTabBench, an 80-task benchmark (58 routine and 22 natural-shift tasks, plus eight stress families) built from these findings without using model scores. Empirically, TabICL and TabPFN lead overall. However, CatBoost leads on natural shift when failed runs are counted, the foundation models lose their lead under strong numerical noise, and 17 of 132 reporting settings change the top-ranked model.

Critical strengths

Strength: The audit finds real problems and the resulting benchmark reports results honestly.

  1. Structured auditing uncovers issues that benchmark descriptions hide, namely test-label leakage into TabFSBench’s stress construction and substitute-model scores in BeyondArena’s aggregate (Sec. 4.1). Both directly change AutoTabBench’s design (Sec. 5.1; Table 6).
  2. The evaluation keeps negative results visible rather than hiding them. Every cell has a recorded outcome, and for example reporting LimiX goes from best to near-worst on Natural Shift in rank (Table 2).
  3. Score-blind task selection (App. A.2), full per-task losses (Figs. 6–8), and the weighting sensitivity analysis (Table 24) make the results checkable and back the claim that conclusions depend on evaluation scope.
Critical weaknesses

Weakness 1: The evaluation of BenchScope is small and graded by the authors.

  1. The paper states that “the author” evaluates all 36 outputs (Sec. 4.2; Appendix B.3), and the references are described only as “author-verified” and “human-verified” (Appendix B.2–B.3). No second grader or agreement rate is reported. How was blinding enforced, given that Skill outputs may use the indicator vocabulary?
  2. The main improvement of 80.56% to 94.44% compares outputs against “human-verified” reference profiles (Appendix B.2). The paper does not say who built these profiles or whether they were fixed before the runs. The claim-level checklist is stated to be fixed in advance (Appendix B.3). The profile references are not.
  3. The paper reports no variance, confidence interval, or significance test for the 80.56% → 94.44% gain, and does not state the LLM sampling settings (Appendix B.1). With three runs, one model, and six benchmarks (Table 9), the size and generality of the gain are uncertain, even though the direction is consistent.
  4. The Skill bundles two ingredients, indicator definitions and evidence-handling rules (Appendix B.1), and the paper does not test them separately, so the source of the gain is unclear.
  5. The profile gain carries over to the four external benchmarks (79.6% → 94.2%, from Table 8). The claim-level gain mostly does not: on external benchmarks, recognized references rise only from 67/72 to 71/72, and 9 of the 13 additional recognitions come from the two formation sources (Appendix B.4).
  6. The only baseline is the same GPT-5.6 agent with no instructions (Sec. 4.2; Appendix B.1). There is no comparison to a generic audit checklist of similar length, another LLM, or the agent-based auditing methods cited in Related Work (Wang et al., 2026; Tu et al., 2026). The gain may therefore reflect “detailed instructions vs. none” rather than this specific Skill.
  7. Human adjudication is used “where needed”, but the number of interventions is not reported, so the “automated” part of BenchScope is not measured.
  8. Overall, BenchScope is half the contribution and the basis of the title. Without independent references and grading, it’s hard to know how large the Skill’s benefit really is.

Weakness 2: The benchmark results rest on a thin evaluation base

  1. AutoTabBench fixes one train/test split per task before evaluation (Sec. 5.3), and the three seeds appear to vary only model randomness (Appendix F.1). The bootstrap intervals are described as “descriptive,” “post-hoc and pointwise,” over tasks that can share sources (Sec. 6.2; Appendix F.6).
  2. Some per-condition conclusions rest on very few datasets. The claims that LightGBM is best on temporal shift and CatBoost on domain shift come from only five and four tasks (Sec. 6.2). In the per-group breakdown (Figure 3; Table 26), some scores come from a single dataset: LimiX’s temporal score is based on the one temporal task it completed, and the Media, Sports, and Safety domains contain one task each.
  3. Some leader changes reflect model configuration. LimiX runs without retrieval, with caps of fewer than 50,000 training rows and at most ten classes (Appendix F.1). This gives 33 unsupported, 18 OOM, and 15 error cells (Table 21). I understand this is deliberate and that failure-last ranking is designed to expose it. Still, “CatBoost leads Natural Shift” then partly describes this configuration, where LimiX completes 27 of 66 cells (Table 2), rather than LimiX itself.
  4. Seven models are run, all at defaults without tuning, with no neural network trained from scratch (e.g., an MLP or FT-Transformer). Runtime and memory are not reported for the main comparison, even though the audit scores other benchmarks on compute and efficiency disclosure.
  5. Overall, for a benchmark paper the artifact and its statistical base are the contribution. With one split per task, single-digit subgroups, and no released supplement material of the software, it’s hard to verify that the reported leader changes are stable.
Questions for authors

Major questions:

  1. Is the no-Skill baseline a fair comparison? The main BenchScope result (80.56% → 94.44%, Table 1) compares the same GPT-5.6 agent with and without the Boundary Skill. Both get the same papers, code, and tools. The Skill version additionally receives the 44 indicator definitions and evidence-handling rules (Appendix B.1). Answers are graded by exact match to fixed answer options (Table 7), but the paper does not say whether the baseline is shown in these options. If not, part of the gain may come from answer formatting. Can the authors clarify this and add stronger baselines: (i) the baseline given the 20 questions and answer options only, (ii) a generic audit checklist of similar length, (iii) an existing agent-based auditor (Wang et al., 2026; Tu et al., 2026), and (iv) one other LLM?
  2. Who prepared and graded the answers used to evaluate BenchScope? BenchScope’s accuracy is measured against reference answers prepared by the authors: a 20-question profile per benchmark (Table 7) and a 42-item checklist of expected findings. These are described only as “human-verified” and “author-verified” (Appendix B), and the paper states that “the author” graded all 36 outputs (Section 4.2). Who prepared these references, and were they fixed before the agent runs? Two of the 20 questions (fields 18 & 19) match the authors’ own TabFSBench and BeyondArena findings (Section 4.1), although both benchmarks are presented as unseen test cases. Was the Skill finalized before these audits? Can a second grader who did not design the Skill re-score a subset and report agreement (e.g., Cohen’s κ)?
  3. Are the 80 tasks really distinct base identities? Some tasks appear to share source data. Superconduct and superconductivity give nearly identical losses for every model (Figure 7), and the names suggest possible overlap for red wine, white wine, and wine quality, and for video transcoding in Real-Core vs. Natural Shift. Can the authors confirm whether these share base data?
  4. Where does the candidate pool come from, and does selection matter? The final 80 tasks are chosen from only 88 candidates, so all seven candidate collections share at least 72 tasks and give the same leader (Table 5). The pool itself therefore drives the benchmark. How were the “80 previously validated configurations” in it assembled (Sec. 5.2), and were model results seen at that stage? A comparison with a simple alternative, such as TabArena plus the official shift tasks of BeyondArena and TableShift, would show what BenchScope adds.
  5. How stable are the rankings? AutoTabBench fixes one train/test split per task, and the three seeds appear to vary only model randomness. This is expected for Natural Shift, but repeated splits are standard for the 58 Real-Core tasks (e.g., TabArena paper).
    • Do the TabICL-TabPFN gap (Table 2) and the Natural Shift leader change survive split resampling?
    • The intervals are described as descriptive and post-hoc (Appendix F.6). Can the authors report Friedman tests with post-hoc pairwise comparisons, or Wilcoxon signed-rank tests with Holm correction, for Tables 2 and 3 and the condition groups in Sec. 6.2?
  6. Does the Natural Shift conclusion survive another LimiX configuration? LimiX runs without retrieval, with a classification cap below 50,000 training rows and ten classes (Appendix F.1). It completes 27 of 66 Natural Shift cells (Table 2; Table 21).
    • Does "CatBoost leads Natural Shift" still hold if LimiX runs with retrieval, or with explicit, reported subsampling? Why was the no-retrieval setting chosen as the default?
  7. Does the loss ratio measure robustness, or does it favor weaker baselines? The ratio divides stressed loss by unperturbed loss, so a higher baseline loss gives a smaller ratio for the same absolute increase. Table 3 shows ExtraTrees has the second-worst unperturbed rank but the smallest noise ratio, yet Sec. 6.3 says it shows "less degradation."
    • Can the authors add a scale-free measure that doesn't depend on baseline loss, such as the within-task rank of the absolute loss increase?
    • At which noise scale does the TabICL and TabPFN rank drop begin? Only 1× IQR is in the main text, while Figures 9–10 include 0.1× and 0.5×.

Minor questions:

  1. Do the TFM results change with categorical metadata? The foundation models are not told which columns are categorical (Appendix F.1), while Appendix F.4 shows that this choice changes the tree models’ results under stress (Table 20). Can the authors run the same check for TabPFN and TabICL on the unseen-category and noise stresses, and state in Sec. 6.1 why shared encoding is the default?
  2. How much human involvement was needed? What fraction of fields across the 37 entries needed re-inspection or human adjudication (Sec. 3.1), what is agent-only accuracy without it, and what did one audit cost in time and tokens? This bears on how “automated” AutoTabBench is.
  3. Can the protocol indicators detect a flawed but well-documented protocol? E01 to E05 record whether something is disclosed, not whether it is appropriate (Table 13). Would E03 (information parity) have flagged the TabFSBench test-label leak on its own?
  4. Can reviewers access the benchmark now? Code, manifests, and ledgers are promised only “upon acceptance” (footnote 1). Could an anonymized release be provided during discussion, and what is the maintenance plan after v0.4?
  5. Can the authors report fit/predict time and peak GPU memory per model? The paper’s own indicators treat these as part of a benchmark’s scope (Table 13, E05, E09).
  6. Would a broader model set change the leaders? TabPFN v2, TabICL v1, and TabSTAR are cited but not run, and there is no neural network trained from scratch (e.g., an MLP or FT-Transformer).
Reviewer_RgeiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BenchScope, an evidence-grounded framework for analyzing the evaluation scope of tabular benchmarks, and AutoTabBench, a benchmark constructed using the resulting analysis. BenchScope combines agent-assisted information extraction, script-based verification, and 44 rule-based indicators to characterize task coverage, generalization conditions, evaluation protocols, and evidence completeness.

Using this framework, the authors construct AutoTabBench with 58 routine-prediction tasks, 22 natural distribution-shift tasks, and eight controlled stress families. Experiments with seven tabular models demonstrate that model rankings and robustness characteristics vary across evaluation conditions. The authors also report an improvement in benchmark boundary recovery accuracy from 80.56% to 94.44% when using their Boundary Skill.

Critical strengths

1. A practically relevant problem. The paper emphasizes that aggregate benchmark performance does not necessarily reflect robustness across different evaluation conditions. Explicitly documenting benchmark scope, protocol assumptions, and evidence provenance is valuable for interpreting comparisons between tabular foundation models.

2. Useful benchmark auditing results. BenchScope identifies potentially consequential evaluation issues, including test-label-informed feature perturbation and the treatment of failed model executions. The improvement in structured boundary recovery accuracy also suggests that the proposed workflow can facilitate more systematic benchmark auditing.

Critical weaknesses

1. Limited novelty and unclear added value over existing benchmarks.

Most evaluation conditions considered in AutoTabBench, including feature perturbations, missingness, and distribution shifts, have already been extensively studied in benchmarks such as TabFSBench, TableShift, TabReD, and BeyondArena. In particular, BeyondArena already evaluates tabular foundation models across IID, temporal, and grouped settings and demonstrates substantial changes in model rankings.

While BenchScope provides an evidence-grounded workflow for auditing and assembling benchmarks, the paper does not sufficiently demonstrate what additional evaluation capabilities or previously unidentified model limitations AutoTabBench offers beyond existing benchmarks. The distinction between integrating established evaluation settings and introducing genuinely new benchmark insights remains unclear.

2. Insufficient validation of the benefits of automated benchmark construction.

The improvement in boundary recovery accuracy from 80.56% to 94.44% demonstrates more reliable information extraction, but does not directly establish that the resulting benchmark is superior to simpler alternatives, such as manually curated benchmark combinations.

Although the authors analyze task coverage, selection policies, and reporting sensitivity, the evaluation provides limited direct evidence that BenchScope-driven construction produces a more informative or representative benchmark. Furthermore, the composition stability analysis selects 80 tasks from 88 candidates, guaranteeing at least 72 overlapping tasks between any two collections. This limits the conclusions that can be drawn about stability under substantially different benchmark compositions.

Questions for authors

Q1. Added value over prior benchmarks. What evaluation capabilities or previously unidentified model limitations does AutoTabBench reveal beyond existing benchmarks such as BeyondArena, TabFSBench, and TableShift? Could the authors provide a direct comparison of coverage and incremental findings?

Q2. Effectiveness of benchmark construction. Beyond improving evidence recovery accuracy, how does BenchScope demonstrate that its task-selection procedure produces a more informative benchmark than simpler approaches, such as manually curated benchmark combinations?

Q3. Composition stability. Given that 80 tasks are selected from 88 candidates, would the reported stability persist under more substantially different task collections or independent candidate pools?

Reviewer_3BQXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that conclusions from the existing tabular benchmarks only hold under the conditions it actually tests, such as its tasks, splits, and evaluation rules. To address this issue, the paper proposes BenchScope, an LLM-agent workflow that reads a benchmark's paper, code, and configuration files and records these conditions, marking each value as observed, derived, inferred, or unknown. Giving the agent a set of 44 audit rules raises its accuracy on six benchmarks from 80.56% to 94.44%. Using these audits, the authors build AutoTabBench, an 80-task benchmark. Evaluating seven models on it shows that the leading model changes with the test condition and with how scores are averaged.

Critical strengths

Most benchmarks report a single leaderboard. This paper instead re-ranks the models after changing the task set and the way scores are averaged. It also reports failed runs instead of hiding them. For example, TabICL stays first across different task sets, but LightGBM becomes first when larger datasets get more weight. This tells readers which conclusions are stable and which depend on reporting choices.

Critical weaknesses

W1. The methodological contribution amounts to a rule document for an LLM agent. BenchScope's core component, Boundary Skill, is a set of 44 indicator definitions and evidence-handling rules given to a GPT-5.6 agent, and its evaluation only compares the same agent with and without these rules. That supplying explicit rules improves extraction accuracy is trivially expected result. Without comparisons to simpler alternatives (e.g., a plain checklist prompt, other LLMs, or manual auditing effort), it is unclear what is methodologically new beyond a well-engineered agent workflow.

W2. Limited novelty AutoTabBench re-selects existing tasks (32 of the 88 candidates come from TabArena). Its main message, that leading models change with evaluation conditions and aggregation, is almost same with prior observations from benchmark lottery (Dehghani et al., 2021), which the paper itself cites. Condition-level conclusions also rely on very few tasks (e.g., 5 temporal and 4 domain-shift tasks) without any uncertainty estimatations.

Questions for authors

Q. Please provide the full list of the 80 AutoTabBench tasks with their sources. The paper states that 32 of the 88 candidates come from TabArena and 56 from other sources, and that the pool builds on 80 "validated configurations", but neither the origin of these configurations nor a task-level list is given; only the 11 BeyondArena tasks are named (Table 15). A table with each task's dataset, source, target, split, and track would be needed to assess novelty and coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.