- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that benchmarks for tabular foundation models rarely state the conditions under which their conclusions hold, such as which tasks, splits, and evaluation protocols they cover. It makes two contributions.
The first is BenchScope, a GPT-5.6 agent guided by a written “Boundary Skill” of 44 indicators. It reads a benchmark’s paper and code and produces a structured profile of what the benchmark tests, with each value linked to its source and marked as observed, inferred, or unknown. With the Skill, the agent’s accuracy on 20 profile fields rises from 80.56% to 94.44% across six benchmarks. The second is AutoTabBench, an 80-task benchmark (58 routine and 22 natural-shift tasks, plus eight stress families) built from these findings without using model scores. Empirically, TabICL and TabPFN lead overall. However, CatBoost leads on natural shift when failed runs are counted, the foundation models lose their lead under strong numerical noise, and 17 of 132 reporting settings change the top-ranked model.
Critical strengths
Strength: The audit finds real problems and the resulting benchmark reports results honestly.
- Structured auditing uncovers issues that benchmark descriptions hide, namely test-label leakage into TabFSBench’s stress construction and substitute-model scores in BeyondArena’s aggregate (Sec. 4.1). Both directly change AutoTabBench’s design (Sec. 5.1; Table 6).
- The evaluation keeps negative results visible rather than hiding them. Every cell has a recorded outcome, and for example reporting LimiX goes from best to near-worst on Natural Shift in rank (Table 2).
- Score-blind task selection (App. A.2), full per-task losses (Figs. 6–8), and the weighting sensitivity analysis (Table 24) make the results checkable and back the claim that conclusions depend on evaluation scope.
Critical weaknesses
Weakness 1: The evaluation of BenchScope is small and graded by the authors.
- The paper states that “the author” evaluates all 36 outputs (Sec. 4.2; Appendix B.3), and the references are described only as “author-verified” and “human-verified” (Appendix B.2–B.3). No second grader or agreement rate is reported. How was blinding enforced, given that Skill outputs may use the indicator vocabulary?
- The main improvement of 80.56% to 94.44% compares outputs against “human-verified” reference profiles (Appendix B.2). The paper does not say who built these profiles or whether they were fixed before the runs. The claim-level checklist is stated to be fixed in advance (Appendix B.3). The profile references are not.
- The paper reports no variance, confidence interval, or significance test for the 80.56% → 94.44% gain, and does not state the LLM sampling settings (Appendix B.1). With three runs, one model, and six benchmarks (Table 9), the size and generality of the gain are uncertain, even though the direction is consistent.
- The Skill bundles two ingredients, indicator definitions and evidence-handling rules (Appendix B.1), and the paper does not test them separately, so the source of the gain is unclear.
- The profile gain carries over to the four external benchmarks (79.6% → 94.2%, from Table 8). The claim-level gain mostly does not: on external benchmarks, recognized references rise only from 67/72 to 71/72, and 9 of the 13 additional recognitions come from the two formation sources (Appendix B.4).
- The only baseline is the same GPT-5.6 agent with no instructions (Sec. 4.2; Appendix B.1). There is no comparison to a generic audit checklist of similar length, another LLM, or the agent-based auditing methods cited in Related Work (Wang et al., 2026; Tu et al., 2026). The gain may therefore reflect “detailed instructions vs. none” rather than this specific Skill.
- Human adjudication is used “where needed”, but the number of interventions is not reported, so the “automated” part of BenchScope is not measured.
- Overall, BenchScope is half the contribution and the basis of the title. Without independent references and grading, it’s hard to know how large the Skill’s benefit really is.
Weakness 2: The benchmark results rest on a thin evaluation base
- AutoTabBench fixes one train/test split per task before evaluation (Sec. 5.3), and the three seeds appear to vary only model randomness (Appendix F.1). The bootstrap intervals are described as “descriptive,” “post-hoc and pointwise,” over tasks that can share sources (Sec. 6.2; Appendix F.6).
- Some per-condition conclusions rest on very few datasets. The claims that LightGBM is best on temporal shift and CatBoost on domain shift come from only five and four tasks (Sec. 6.2). In the per-group breakdown (Figure 3; Table 26), some scores come from a single dataset: LimiX’s temporal score is based on the one temporal task it completed, and the Media, Sports, and Safety domains contain one task each.
- Some leader changes reflect model configuration. LimiX runs without retrieval, with caps of fewer than 50,000 training rows and at most ten classes (Appendix F.1). This gives 33 unsupported, 18 OOM, and 15 error cells (Table 21). I understand this is deliberate and that failure-last ranking is designed to expose it. Still, “CatBoost leads Natural Shift” then partly describes this configuration, where LimiX completes 27 of 66 cells (Table 2), rather than LimiX itself.
- Seven models are run, all at defaults without tuning, with no neural network trained from scratch (e.g., an MLP or FT-Transformer). Runtime and memory are not reported for the main comparison, even though the audit scores other benchmarks on compute and efficiency disclosure.
- Overall, for a benchmark paper the artifact and its statistical base are the contribution. With one split per task, single-digit subgroups, and no released supplement material of the software, it’s hard to verify that the reported leader changes are stable.
Questions for authors
Major questions:
- Is the no-Skill baseline a fair comparison? The main BenchScope result (80.56% → 94.44%, Table 1) compares the same GPT-5.6 agent with and without the Boundary Skill. Both get the same papers, code, and tools. The Skill version additionally receives the 44 indicator definitions and evidence-handling rules (Appendix B.1). Answers are graded by exact match to fixed answer options (Table 7), but the paper does not say whether the baseline is shown in these options. If not, part of the gain may come from answer formatting. Can the authors clarify this and add stronger baselines: (i) the baseline given the 20 questions and answer options only, (ii) a generic audit checklist of similar length, (iii) an existing agent-based auditor (Wang et al., 2026; Tu et al., 2026), and (iv) one other LLM?
- Who prepared and graded the answers used to evaluate BenchScope? BenchScope’s accuracy is measured against reference answers prepared by the authors: a 20-question profile per benchmark (Table 7) and a 42-item checklist of expected findings. These are described only as “human-verified” and “author-verified” (Appendix B), and the paper states that “the author” graded all 36 outputs (Section 4.2). Who prepared these references, and were they fixed before the agent runs? Two of the 20 questions (fields 18 & 19) match the authors’ own TabFSBench and BeyondArena findings (Section 4.1), although both benchmarks are presented as unseen test cases. Was the Skill finalized before these audits? Can a second grader who did not design the Skill re-score a subset and report agreement (e.g., Cohen’s κ)?
- Are the 80 tasks really distinct base identities? Some tasks appear to share source data. Superconduct and superconductivity give nearly identical losses for every model (Figure 7), and the names suggest possible overlap for red wine, white wine, and wine quality, and for video transcoding in Real-Core vs. Natural Shift. Can the authors confirm whether these share base data?
- Where does the candidate pool come from, and does selection matter? The final 80 tasks are chosen from only 88 candidates, so all seven candidate collections share at least 72 tasks and give the same leader (Table 5). The pool itself therefore drives the benchmark. How were the “80 previously validated configurations” in it assembled (Sec. 5.2), and were model results seen at that stage? A comparison with a simple alternative, such as TabArena plus the official shift tasks of BeyondArena and TableShift, would show what BenchScope adds.
- How stable are the rankings? AutoTabBench fixes one train/test split per task, and the three seeds appear to vary only model randomness. This is expected for Natural Shift, but repeated splits are standard for the 58 Real-Core tasks (e.g., TabArena paper).
- Do the TabICL-TabPFN gap (Table 2) and the Natural Shift leader change survive split resampling?
- The intervals are described as descriptive and post-hoc (Appendix F.6). Can the authors report Friedman tests with post-hoc pairwise comparisons, or Wilcoxon signed-rank tests with Holm correction, for Tables 2 and 3 and the condition groups in Sec. 6.2?
- Does the Natural Shift conclusion survive another LimiX configuration? LimiX runs without retrieval, with a classification cap below 50,000 training rows and ten classes (Appendix F.1). It completes 27 of 66 Natural Shift cells (Table 2; Table 21).
- Does "CatBoost leads Natural Shift" still hold if LimiX runs with retrieval, or with explicit, reported subsampling? Why was the no-retrieval setting chosen as the default?
- Does the loss ratio measure robustness, or does it favor weaker baselines? The ratio divides stressed loss by unperturbed loss, so a higher baseline loss gives a smaller ratio for the same absolute increase. Table 3 shows ExtraTrees has the second-worst unperturbed rank but the smallest noise ratio, yet Sec. 6.3 says it shows "less degradation."
- Can the authors add a scale-free measure that doesn't depend on baseline loss, such as the within-task rank of the absolute loss increase?
- At which noise scale does the TabICL and TabPFN rank drop begin? Only 1× IQR is in the main text, while Figures 9–10 include 0.1× and 0.5×.
Minor questions:
- Do the TFM results change with categorical metadata? The foundation models are not told which columns are categorical (Appendix F.1), while Appendix F.4 shows that this choice changes the tree models’ results under stress (Table 20). Can the authors run the same check for TabPFN and TabICL on the unseen-category and noise stresses, and state in Sec. 6.1 why shared encoding is the default?
- How much human involvement was needed? What fraction of fields across the 37 entries needed re-inspection or human adjudication (Sec. 3.1), what is agent-only accuracy without it, and what did one audit cost in time and tokens? This bears on how “automated” AutoTabBench is.
- Can the protocol indicators detect a flawed but well-documented protocol? E01 to E05 record whether something is disclosed, not whether it is appropriate (Table 13). Would E03 (information parity) have flagged the TabFSBench test-label leak on its own?
- Can reviewers access the benchmark now? Code, manifests, and ledgers are promised only “upon acceptance” (footnote 1). Could an anonymized release be provided during discussion, and what is the maintenance plan after v0.4?
- Can the authors report fit/predict time and peak GPU memory per model? The paper’s own indicators treat these as part of a benchmark’s scope (Table 13, E05, E09).
- Would a broader model set change the leaders? TabPFN v2, TabICL v1, and TabSTAR are cited but not run, and there is no neural network trained from scratch (e.g., an MLP or FT-Transformer).