- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TabFM-Auto, which pairs an LLM coding agent with a frozen 400M-parameter tabular foundation model (TabFM).
Method : The agent edits only the data pipeline around the model: four Python hooks (cleaning and target transforms, feature engineering, context-row selection, post-processing and calibration) plus TabFM’s constructor settings. It is guided by dataset metadata and by 3-fold cross-validation on the training split of TabArena’s fold 0, with up to 96 evaluations or 6 H100-hours per dataset.
TabArena results : Evaluated on all official folds of the 51 datasets, the five configurations occupy the top five Elo positions, and the best raises TabFM from 1785 to 2013 Elo.
Transfer and ablation : The discovered pipelines transfer to TabPFN-3, TabICLv2 and EXAONE-Tabular (+69 to +143 Elo). An unconstrained agent with the same budget but without TabFM reaches only 1469 Elo.
MLE-Bench : On the 8 tabular competitions, the Opus 5 configuration ranks first among 14 agents and earns four medals.
Critical strengths
I see two major strengths for this paper :
-
A clean, practically relevant design supported by broad evidence. Freezing the foundation model and restricting the agent to readable pipeline code makes candidate evaluation cheap and attributes every change to the pipeline. The evidence spans 51 TabArena datasets, transfer to three other foundation models, and eight MLE-Bench competitions whose test labels are hidden from the agent. In addition, the same agent without TabFM is 511 Elo lower (Table 2)
-
Interpretable outputs and informative analysis. Representative pipelines are shown as short readable code (Fig. 3). The paper also analyzes which stages the agents modify and where gains concentrate (Sec. 4.1), and reports token and tool usage across harnesses (Table 3). Transfer to other foundation models suggests that the discovered knowledge is not purely TabFM-specific. This is consequential because readable, transferable pipelines are what could make the approach reusable beyond leaderboard gains.
Critical weaknesses
The methodological contribution is incremental, and the empirical insights that would justify it are not established. Each building block has a direct precedent, several of them acknowledged in the paper itself.
Two claims are form my side not supported by experiments :
-
The claim that LLM-written pipelines supply “the semantics missing from synthetic pretraining” (Sec. 5) : no clear evidence is shown about the fact that
-
The “core claim that coding agents perform better when paired with a frozen foundation model” (Sec. 4.1) : no experiments were made with finetuning/postuning the model.
No uncertainty and no replication. Table 1 and Fig. 1 report point Elo ratings without intervals. Each configuration is searched once per dataset, and the headline +228 Elo is the best of five configurations.
Moreover, the main value for the paper for me: it suggests that part of the remaining error of tabular foundation models can be removed by data-pipeline work rather than by new architectures. Some papers ( eg, TFM-Retouche) shows that adapting data input to a task improves the performance of the state-of-the-art models.
Questions for authors
How does TabFM-Auto compare with each of the following?
- HPO over TabFM( TABFM_KWARGS in the paper) and standard preprocessing at equal budget.
- CAAFE, FeatLLM or OCTree with TabFM as the predictor.
- The same agents with anonymized column names and no task description.
6 comments
Following Up
Hi Reviewer b2py,
Has this rebuttal changed your mind in any way? Can you please respond?
Regards, AC
Thank you for the rebuttal and the new experiments. I acknowledge the bootstrap intervals in Tables 4–5, which I had missed, and I accept the clarification on “frozen” (W2b). My main concerns remain:
-
Clean evaluation. Fold 0 is the only split whose test rows were not used during search. Every row is a test row once per repetition, so two thirds of all-folds test predictions are on rows whose labels guided the search. On fold 0 (Table 5), TabFM+ ranks above three of the five configurations, and on classification two configurations fall below zero-shot TabFM. Fold-0 estimates are noisy, but the “top five” result does not hold on the only clean split. The overfitting analysis of Fig. 4 and 15 also relies on folds that overlap the search data.
-
Semantics (W2a/Q3). The anonymization ablation covers four datasets, with one run each, using a configuration absent from the paper. On three of the four datasets, the gap (0.45–0.67 points) is below the 0.72% median rerun difference you report. Even at face value, 60% of the gain survives without any schema. The claim that the pipelines supply “the semantics missing from synthetic pretraining” should be narrowed accordingly.
-
Baselines (Q1/Q2). TabFM+ is a fixed preset, not an equal-budget search over TABFM_KWARGS and preprocessing. CAAFE, designed with TabPFN as its downstream model, is set aside by a scope argument, without results. The novelty concern (W1) therefore stands.
-
Table 2. After the W2b clarification, 62% of the 510.8 Elo gap comes from zero-shot TabFM alone. The benefit of restricting the agent to the pipeline remains untested.
For these reasons, I maintain my score.
Thank you for the detailed review and constructive suggestions.
W1. Methodological contribution and empirical insights. TabFM-Auto works because it pairs two complementary strengths: LLMs and coding agents provide world knowledge and flexible pipeline code (turning column names, units, domain formulas, and external files into explicit transforms), while TabFM and other TFMs provide strong, training-free numerical prediction on tables (where synthetic pretraining gives strong numerical priors, though without awareness of column semantics). Across 51 TabArena datasets, this pairing lowers G-Mean test error from 0.3550 (TabFM) to 0.3360 (+5.35%) on Antigravity (Gemini 4 Argon) and 0.3400 (+4.23%, 2013.0 Elo #1) on Codex (Opus 5), and ranks #1 of 14 agents on MLE-Bench-Tabular (1827 Elo, Figure 7, p. 9).
Our ablations isolate why both halves matter:
W2a & Q3. Evidence for "the semantics missing from synthetic pretraining" and anonymized-column ablation. Across all 51 datasets under Claude Code (Opus 5) (Table 6, p. 17), the 17 domain-schema datasets (Category I) show a 3.1× larger mean error reduction (+7.25%) than the 34 generic or anonymized datasets (Category II, +2.31%). To test Q3 directly, we ran a controlled ablation on Antigravity (Gemini 4 Argon), replacing column names with
col_0,col_1, ... and removing dataset names and descriptions under identical splits, budget, and frozen TabFM:Removing column names and descriptions reduces gains by 30% to 47% (40% on average) on tasks with domain formulas, while statistical cleaning, relational features, sampling, and calibration retain the remaining 60%.
W2b. Clarifying "frozen foundation model" (Section 4.1). In Section 4.1 (p. 7), "frozen" was meant to contrast keeping TabFM fixed without gradient updates against MLE agents that train models from scratch (Table 2 & Figure 7), not against fine-tuning TabFM. We will reword this clearly.
W3. Uncertainty intervals and consistency across sweeps. Tables 4 & 5 (pp. 16-17) report 95% bootstrap CIs (for example, Codex + Opus 5 at 2013.0 [−103.2,+123.7] vs. AutoGluon 1.5 at 1668.4 [−79.2,+80.6]), which we will add to Figure 1 and Table 1. Because Elo shifts when baseline ratings change and treats narrow and wide margins equally, we also compare 51-dataset G-Mean test error (↓) across all six sweeps (3 harnesses and 3 LLMs; an independent rerun of Antigravity + Gemini 3.8 Flash also agrees within 0.72% median difference):
W4. TFM-Retouche. We will cite TFM-Retouche in Section 2. It trains numerical input adapters via gradients without reading column names, while TabFM-Auto writes gradient-free Python pipelines from task metadata, so the two approaches are complementary.
Questions 1-2 (Q3 is answered in W2a above).
TABFM_KWARGSand preprocessing: TabFM+ (0.3513 G-Mean, 1856.0 Elo) enables TabFM's normalization, feature crosses, SVD, and view ensembling, reducing error by +1.06% vs. +4.19% to +5.35% for Opus 5 and Gemini 4 Argon. None of the 51 Gemini 4 Argon runs tuneTABFM_KWARGSalone, and our TFM transfer runs (+88.7 to +143.3 Elo, Table 7) dropTABFM_KWARGS.