acceptodds
Under review as a conference paper at ICLR 2027

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

13 views

Abstract

Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. In parallel, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM’s error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. Moreover, the discovered pipelines transfer to other frozen tabular foundation models (+89 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation model, self improving agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_b2pyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TabFM-Auto, which pairs an LLM coding agent with a frozen 400M-parameter tabular foundation model (TabFM).

Method : The agent edits only the data pipeline around the model: four Python hooks (cleaning and target transforms, feature engineering, context-row selection, post-processing and calibration) plus TabFM’s constructor settings. It is guided by dataset metadata and by 3-fold cross-validation on the training split of TabArena’s fold 0, with up to 96 evaluations or 6 H100-hours per dataset.

TabArena results : Evaluated on all official folds of the 51 datasets, the five configurations occupy the top five Elo positions, and the best raises TabFM from 1785 to 2013 Elo.

Transfer and ablation : The discovered pipelines transfer to TabPFN-3, TabICLv2 and EXAONE-Tabular (+69 to +143 Elo). An unconstrained agent with the same budget but without TabFM reaches only 1469 Elo.

MLE-Bench : On the 8 tabular competitions, the Opus 5 configuration ranks first among 14 agents and earns four medals.

Critical strengths

I see two major strengths for this paper :

  • A clean, practically relevant design supported by broad evidence. Freezing the foundation model and restricting the agent to readable pipeline code makes candidate evaluation cheap and attributes every change to the pipeline. The evidence spans 51 TabArena datasets, transfer to three other foundation models, and eight MLE-Bench competitions whose test labels are hidden from the agent. In addition, the same agent without TabFM is 511 Elo lower (Table 2)

  • Interpretable outputs and informative analysis. Representative pipelines are shown as short readable code (Fig. 3). The paper also analyzes which stages the agents modify and where gains concentrate (Sec. 4.1), and reports token and tool usage across harnesses (Table 3). Transfer to other foundation models suggests that the discovered knowledge is not purely TabFM-specific. This is consequential because readable, transferable pipelines are what could make the approach reusable beyond leaderboard gains.

Critical weaknesses

The methodological contribution is incremental, and the empirical insights that would justify it are not established. Each building block has a direct precedent, several of them acknowledged in the paper itself.

Two claims are form my side not supported by experiments :

  • The claim that LLM-written pipelines supply “the semantics missing from synthetic pretraining” (Sec. 5) : no clear evidence is shown about the fact that

  • The “core claim that coding agents perform better when paired with a frozen foundation model” (Sec. 4.1) : no experiments were made with finetuning/postuning the model.

No uncertainty and no replication. Table 1 and Fig. 1 report point Elo ratings without intervals. Each configuration is searched once per dataset, and the headline +228 Elo is the best of five configurations.

Moreover, the main value for the paper for me: it suggests that part of the remaining error of tabular foundation models can be removed by data-pipeline work rather than by new architectures. Some papers ( eg, TFM-Retouche) shows that adapting data input to a task improves the performance of the state-of-the-art models.

Questions for authors

How does TabFM-Auto compare with each of the following?

  • HPO over TabFM( TABFM_KWARGS in the paper) and standard preprocessing at equal budget.
  • CAAFE, FeatLLM or OCTree with TabFM as the predictor.
  • The same agents with anonymized column names and no task description.
6 comments

Thank you for the detailed review and constructive suggestions.

W1. Methodological contribution and empirical insights. TabFM-Auto works because it pairs two complementary strengths: LLMs and coding agents provide world knowledge and flexible pipeline code (turning column names, units, domain formulas, and external files into explicit transforms), while TabFM and other TFMs provide strong, training-free numerical prediction on tables (where synthetic pretraining gives strong numerical priors, though without awareness of column semantics). Across 51 TabArena datasets, this pairing lowers G-Mean test error from 0.3550 (TabFM) to 0.3360 (+5.35%) on Antigravity (Gemini 4 Argon) and 0.3400 (+4.23%, 2013.0 Elo #1) on Codex (Opus 5), and ranks #1 of 14 agents on MLE-Bench-Tabular (1827 Elo, Figure 7, p. 9).

Our ablations isolate why both halves matter:

  • Removing world knowledge via schema anonymization cuts domain-task gains by 40% on average (W2a & Q3).
  • Removing TabFM so the same agent trains GBDTs and neural networks drops performance by 510.8 Elo (0.3371 vs. 0.4400 G-Mean, Table 2), while zero-gradient TFM inference keeps Fold-0 (0.3310) and All-Splits (0.3360) G-Mean consistent.
  • Transferring the discovered Python hooks unchanged to three other frozen TFMs adds +88.7 to +143.3 Elo without re-search (Figure 5 & Table 7).

W2a & Q3. Evidence for "the semantics missing from synthetic pretraining" and anonymized-column ablation. Across all 51 datasets under Claude Code (Opus 5) (Table 6, p. 17), the 17 domain-schema datasets (Category I) show a 3.1×3.1\times larger mean error reduction (+7.25%) than the 34 generic or anonymized datasets (Category II, +2.31%). To test Q3 directly, we ran a controlled ablation on Antigravity (Gemini 4 Argon), replacing column names with col_0, col_1, ... and removing dataset names and descriptions under identical splits, budget, and frozen TabFM:

Dataset (Antigravity, Gemini 4 Argon)Full-Schema GainAnonymized-Schema GainGain Retained
diabetes+3.74%+2.60%69.5%
miami_housing+1.61%+0.97%60.1%
houses+1.58%+0.91%57.7%
diamonds+0.95%+0.50%52.7%
Mean (domain-formula tasks)+1.97%+1.24%60.0%

Removing column names and descriptions reduces gains by 30% to 47% (40% on average) on tasks with domain formulas, while statistical cleaning, relational features, sampling, and calibration retain the remaining 60%.

W2b. Clarifying "frozen foundation model" (Section 4.1). In Section 4.1 (p. 7), "frozen" was meant to contrast keeping TabFM fixed without gradient updates against MLE agents that train models from scratch (Table 2 & Figure 7), not against fine-tuning TabFM. We will reword this clearly.

W3. Uncertainty intervals and consistency across sweeps. Tables 4 & 5 (pp. 16-17) report 95% bootstrap CIs (for example, Codex + Opus 5 at 2013.0 [−103.2,+123.7][-103.2, +123.7] vs. AutoGluon 1.5 at 1668.4 [−79.2,+80.6][-79.2, +80.6]), which we will add to Figure 1 and Table 1. Because Elo shifts when baseline ratings change and treats narrow and wide margins equally, we also compare 51-dataset G-Mean test error (↓\downarrow) across all six sweeps (3 harnesses and 3 LLMs; an independent rerun of Antigravity + Gemini 3.8 Flash also agrees within 0.72% median difference):

MethodAll-Splits G-Mean ↓\downarrowGain vs. TabFMAll-Splits Elo ↑\uparrow
Antigravity, Gemini 4 Argon (new)0.3360+5.35%1969.5
Antigravity, Gemini 3.8 Flash0.3371+5.05%1979.6
Codex, Opus 50.3400+4.23%2013.0
Claude Code, Opus 50.3402+4.19%1993.6
Codex, Gemini 3.8 Flash0.3435+3.26%1940.1
Claude Code, Gemini 3.8 Flash0.3446+2.93%1957.8
TabFM+ (no search)0.3513+1.06%1856.0
TabFM (zero-shot)0.35500.00%1785.3

W4. TFM-Retouche. We will cite TFM-Retouche in Section 2. It trains numerical input adapters via gradients without reading column names, while TabFM-Auto writes gradient-free Python pipelines from task metadata, so the two approaches are complementary.

Questions 1-2 (Q3 is answered in W2a above).

  1. HPO over TABFM_KWARGS and preprocessing: TabFM+ (0.3513 G-Mean, 1856.0 Elo) enables TabFM's normalization, feature crosses, SVD, and view ensembling, reducing error by +1.06% vs. +4.19% to +5.35% for Opus 5 and Gemini 4 Argon. None of the 51 Gemini 4 Argon runs tune TABFM_KWARGS alone, and our TFM transfer runs (+88.7 to +143.3 Elo, Table 7) drop TABFM_KWARGS.
  2. CAAFE, FeatLLM, and OCTree: Those methods only append columns to a single table, while TabFM-Auto also calibrates outputs, cleans sentinels and targets, samples balanced contexts, and joins external waveform, GNSS, and 3D geometry files on MLE-Bench-Tabular (Table 8, p. 26).
Area_Chair_N6DmOpenReview ↗

Following Up

Hi Reviewer b2py,

Has this rebuttal changed your mind in any way? Can you please respond?

Regards, AC

Thank you for the rebuttal and the new experiments. I acknowledge the bootstrap intervals in Tables 4–5, which I had missed, and I accept the clarification on “frozen” (W2b). My main concerns remain:

  • Clean evaluation. Fold 0 is the only split whose test rows were not used during search. Every row is a test row once per repetition, so two thirds of all-folds test predictions are on rows whose labels guided the search. On fold 0 (Table 5), TabFM+ ranks above three of the five configurations, and on classification two configurations fall below zero-shot TabFM. Fold-0 estimates are noisy, but the “top five” result does not hold on the only clean split. The overfitting analysis of Fig. 4 and 15 also relies on folds that overlap the search data.

  • Semantics (W2a/Q3). The anonymization ablation covers four datasets, with one run each, using a configuration absent from the paper. On three of the four datasets, the gap (0.45–0.67 points) is below the 0.72% median rerun difference you report. Even at face value, 60% of the gain survives without any schema. The claim that the pipelines supply “the semantics missing from synthetic pretraining” should be narrowed accordingly.

  • Baselines (Q1/Q2). TabFM+ is a fixed preset, not an equal-budget search over TABFM_KWARGS and preprocessing. CAAFE, designed with TabPFN as its downstream model, is set aside by a scope argument, without results. The novelty concern (W1) therefore stands.

  • Table 2. After the W2b clarification, 62% of the 510.8 Elo gap comes from zero-shot TabFM alone. The benefit of restricting the agent to the pipeline remains untested.

For these reasons, I maintain my score.

Reviewer_vwrJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes TabFM-Auto, which uses an LLM coding agent to search for a data pipeline around a frozen Tabular Foundation Model, including data cleaning, semantic feature engineering, context selection, and prediction post-processing, thereby addressing the problem that traditional Tabular Foundation Models cannot directly exploit column names and task metadata. Through experiments, it demonstrates the effectiveness of combining an LLM agent with a frozen tabular foundation model, and verifies the contribution of pipeline search through ablation and test-isolation analyses.

Critical strengths
  1. The experiments have broad coverage. The paper conducts experiments on 51 TabArena datasets and 8 MLE-Bench-Tabular competitions, covering classification, regression, and more complex competition scenarios. This makes me think the method has good generality.
  2. It provides a relatively in-depth analysis of the agent's search behavior. It analyzes what pipelines the agent actually discovered, including domain-specific formulas, statistical/structural features, context selection, and calibration. These experiences can in turn guide human experts or other agents in design, further enhancing the generality of the method.
Critical weaknesses
  1. The main TabArena experiments have cross-fold overlap, which affects the interpretation of "held-out generalization." This severely affects my confidence in the method's actual performance.
  2. There is already a large body of work on LLM + AutoML, and the authors have not sufficiently clarified the distinction from these works. This affects my judgment of the method's novelty.
Questions for authors
  1. I hope the authors can conduct some experiments without overlap to demonstrate the method's true effectiveness (e.g., fold-0-only experiments).
  2. This paper seems to rely heavily on TabFM. Is it still effective on other tabular foundation models?
  3. Some comparison configurations are still missing, such as a TabFM baseline that only tunes TABFM_KWARGS.
  4. Add experiments or analyses related to search budget sensitivity.
3 comments

Thank you for the positive review and helpful questions.

W1 & Q1. Overlap-free (Fold-0) experiments. Our paper reports the overlap-free Fold-0 evaluation in Section 4.1 (pp. 5 & 7), Table 5 (p. 17), and Appendix B.1 (pp. 18-19), and we will move these results into the main text next to Table 1. Because Fold-0 Elo recomputes baseline Elo scores on 51 single splits and swings noticeably on a few flips near 95% win rates while ignoring margin size, we report 51-dataset Geometric Mean test error (G-Mean ↓\downarrow) alongside Elo and also ran a new 51-dataset Antigravity (Gemini 4 Argon) sweep (General Response Table 1):

  • On Fold 0 (zero overlap with search), Antigravity (Gemini 4 Argon) achieves the lowest G-Mean error (0.3310, +6.14% over TabFM, #2 Fold-0 Elo at 1896.2), followed by Codex (Opus 5) at 0.3399 (#1 Fold-0 Elo at 1950.7) and Claude Code (Opus 5) at 0.3413.
  • All six TabFM-Auto sweeps achieve lower Fold-0 G-Mean error (0.3310 to 0.3471) than TabFM+ (0.3501), TabFM (0.3526), EXAONE-Tabular (0.3639), and AutoGluon 1.5 (0.3639).
  • On single-split MLE-Bench-Tabular with hidden test labels (Figure 7, p. 9; Table 8, p. 26), TabFM-Auto (Opus 5) also ranks #1 of 14 agents (1827 Elo, 4 medals).

W2. Distinction from prior LLM + AutoML work. TabFM-Auto pairs the world knowledge and code synthesis of LLM coding agents with the strong, training-free predictive power of TFMs. Prior prompt-based feature methods (CAAFE, FeatLLM, OCTree) only append derived columns to a single table, while end-to-end MLE agents (AIDE, R&D-Agent, MLEvolve) retrain GBDTs or neural networks at each step, which adds CV noise and drops performance by 510.8 Elo at equal budget (Table 2, p. 7). Keeping a strong TFM fixed lets the agent apply domain knowledge and statistical checks across four pipeline stages. We will clarify this distinction in Section 2:

Pipeline StagePython HookClaude Code, Opus 5Antigravity, Gemini 4 Argon (new)Role Around Frozen TabFM
Feature engineeringengineer()98.0%86.3%Domain formulas, ratios, group stats, external files
Output calibrationpostprocess()41.2%58.8%Prior alignment, temperature scaling, inverse links
Cleaning & targetspreprocess()33.3%29.4%Sentinel recoding and reversible target transforms
Context selectionsample()25.5%11.8%Stratified and minority-oversampled context views

Q2. Is TabFM-Auto effective on other tabular foundation models? Yes. Because this pairing relies on the predictive power of in-context TFMs generally rather than TabFM-specific internals, running the four Python hooks discovered by Claude Code (Opus 5) unchanged and without re-search improves three other frozen TFMs across all 51 datasets (Figure 5, p. 7; Table 7, p. 21):

Frozen TFMDefault EloWith Opus 5 PipelinesOverall GainClassification GainRegression Gain
TabICLv21593.01736.7+143.3+116.8+279.5
TabPFN-31666.01797.3+130.8+106.5+318.5
EXAONE-Tabular1773.61862.0+88.7+46.4+225.8

Q3. Comparison with tuning TABFM_KWARGS only. Three results show that the gains come from pipeline code rather than TABFM_KWARGS: (1) TabFM+ (0.3513 G-Mean, 1856.0 Elo) already turns on TabFM's built-in wrapper presets (normalization, feature crosses, SVD, view ensembling, and NNLS weighting) and reduces error by +1.06%, compared to +4.19% to +5.35% for Opus 5 and Gemini 4 Argon; (2) our TFM transfer experiments above (+88.7 to +143.3 Elo) drop TABFM_KWARGS and run only the four Python hooks; and (3) 0 of 51 Gemini 4 Argon runs tune TABFM_KWARGS alone.

Q4. Search budget sensitivity. Figure 4 (p. 6) and Appendix C (Figures 14-15, pp. 23-24) evaluate intermediate checkpoints across budget fractions on all 51 datasets relative to the unedited starting pipeline (0%). Claude Code (Opus 5) passes TabFM+ (1856.0 Elo) within the first 2.5% of search (about 3 evaluations) and continues improving through 100% budget:

Claude Code, Opus 5 (51 Datasets, Fig. 4)0% (Start)2.5%11.5%22.5%56.0%100% Budget
Official Test Error Reduction0.00%+0.71%+1.34%+1.69%+2.96%+3.95%
Official Test TabArena Elo1784.01885.21930.51953.81977.11993.6

Hi Reviewer vwrJ,

Has this rebuttal changed your mind in any way? Can you please respond?

Regards, AC

Dear Reviewer vwrJ,

As the discussion phase is about to end, we kindly ask if our latest experiments and ablations further address your concerns. Again, we thank your thoughtful and detailed reviews!

Best,

Authors

Reviewer_2U3SOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an agentic framework around the tabular foundation model TabFM. The agentic framework uses LLM coding agents to modify static code that does preprocessing, feature engineering, (row) sampling, and post-processing in an iterative loop, driven by cross-validation signals. The method is evaluated on TabArena and MLE-Bench using several LLMs and TFMs.

Critical strengths

The presentation and execution of the proposed idea are excellent. The work is easy to consume and very clear. Moreover, the authors implement the idea straightforwardly, and they present the evaluations clearly. This work is one of the most well-executed and believable on agentic systems for tabular data, and I believe it sets a commendable standard compared to prior agentic systems.

Critical weaknesses

The work suffers from weaknesses in its evaluations, several of which are common to other agentic systems that rarely generalize beyond their own papers.

First, the work lacks a fully valid comparison to baselines on its main benchmark, TabArena. TabFM-Auto is not evaluated for leakage or memorization, which is known to be problematic for tabular data [1,2]. Moreover, TabFM-Auto uses transductive learning but is compared only to inductive methods (TabArena only evaluates inductive learning), giving it a major advantage over baselines. No transductive baselines have been evaluated, and it is questionable if transductive learning would even make sense as a use case in the real world. Lastly, the paper provides no code for the full end-to-end pipeline, nor does it share the discovered pipelines. All of these points together make it hard to estimate how well this work would generalize beyond the paper. And there is no scientific investigation into these major problems of agentic tabular systems in this paper.

Second, the work lacks proper ablations and baseline investigations. The work in Table 2 and Figure 5 goes in the right direction but is too immature. How does a non-pre-prompted coding agent perform with TabFM alone? Why does TabFM matter here? Wouldn't a better metric be the agentic framework's overall improvement across models, including non-TFMs? The paper lacks deeper scientific investigations beyond trying to sell the TabFM + coding agent combination.

[1] Bordt, Sebastian, et al. "Elephants never forget: Memorization and learning of tabular data in large language models." arXiv preprint arXiv:2404.06209 (2024).

[2] Gorla, Aditya, and Ratish Puduppully. "The illusion of generalization: Re-examining tabular language model evaluation." arXiv e-prints (2026): arXiv-2602.

Questions for authors

How do you explain the Elo difference for the TabFM-Auto variants between Table 4 and Table 5? As expected and communicated in the TabArena paper, the first fold is highly representative of all non-LLM models across all splits. But all -auto variants suddenly jump in performance and break the ranking correlation to the first fold. Why would this happen? Especially if the pipeline should be best on the first fold, as it was fit/learned on the first fold. Can you explain this artifact?

Have you investigated whether the pipeline from the first fold includes cheating? Did you investigate the 51 pipeline to see that it did not just "remember" the training data or similar?

7 comments

Thank you for the detailed review and for your kind words on the clarity and execution of our paper.

Q1. Why Elo differs between Table 4 (all splits) and Table 5 (Fold 0). Two details of how Elo is computed explain the difference between Table 4 and Table 5:

  1. Recomputing baseline Elo on a single split: Table 5 recomputes the Elo scores of all 66 baselines from 51 Fold-0 splits instead of 816 splits, fits each TabFM-Auto run separately, and includes 4 baselines evaluated on only a subset of datasets (for example, Mitra covers 33 of 51 datasets). Because the baseline pool itself shifts on a single split, even zero-search TabFM+ on regression moves from 2169.2 in Table 1b to 2409.9 in Table 5.
  2. Sensitivity at high win rates: Elo counts a 0.1% win and a 50% win equally, and when win rates against the pool sit near 95%, a few single-split flips cause large Elo swings.

Reporting 51-dataset Geometric Mean test error (G-Mean ↓\downarrow, which does not depend on the baseline pool) alongside Elo, together with our new Antigravity (Gemini 4 Argon) sweep, shows that held-out performance does not degrade on Fold 0:

MethodFold-0 G-Mean ↓\downarrowAll-Splits G-Mean ↓\downarrowFold-0 Elo ↑\uparrowAll-Splits Elo ↑\uparrow
Antigravity, Gemini 4 Argon (new)0.33100.33601896.21969.5
Codex, Opus 50.33990.34001950.72013.0
Claude Code, Opus 50.34130.34021843.31993.6
Antigravity, Gemini 3.8 Flash0.34450.33711791.91979.6
TabFM+ (no search)0.35010.35131853.21856.0
TabFM (zero-shot)0.35260.35501790.21785.3

All three frontier sweeps have nearly identical G-Mean on Fold 0 and All Splits, and Gemini 4 Argon gains slightly more on Fold 0 (+6.14%) than on All Splits (+5.35%). We will report G-Mean alongside Elo in Table 5.

Q2. Did we check the Fold-0 pipelines for cheating or memorizing training data? Yes. During search, test splits are unmounted inside an offline sandbox and classification rows are shuffled (Appendix A, p. 15). We also inspected all 306 discovered pipeline.py files and confirmed that none access the network, files outside the workspace, row indices, or test labels, and none hardcode target arrays. Only the Python code in pipeline.py transfers from Fold 0: on every fold, statistics are fit on that fold's own training split, and frozen TabFM makes all predictions.

W1. Row memorization [1, 2], transductive learning, and code release.

  • Row memorization [1, 2]: Bordt et al. [1] and Gorla & Puduppully [2] study LLMs predicting labels directly from serialized rows. TabFM-Auto never calls an LLM at test time and never shows test rows to the LLM: the agent writes column-level Python code, and TabFM predicts.
  • Inductive check: 288 of 306 pipelines (94.1%; 49 of 51 in Gemini 4 Argon) are strictly inductive. Reverting the remaining pipelines to zero-shot TabFM keeps every sweep ahead of all baselines:
100% Inductive Method (51 Datasets)Fold-0 G-Mean ↓\downarrowAll-Splits G-Mean ↓\downarrowGain vs. TabFMElo ↑\uparrow
Antigravity, Gemini 4 Argon (new)0.33360.3379+4.83%1955.8
Codex, Opus 50.34300.3426+3.50%1965.5
Claude Code, Opus 50.34410.3430+3.37%1955.5
Antigravity, Gemini 3.8 Flash0.34880.3405+4.08%1932.3
TabFM+0.35010.3513+1.06%1856.0
TabFM0.35260.35500.00%1785.3
AutoGluon 1.5 (4h)0.36390.3718$-$4.74%1668.4
  • Code release: Claude Code, Codex, and Antigravity are public tools; upon institutional approval, we will release our sandboxed evaluator, prompts, per-fold results, and all 306 pipelines.

W2. Ablations and why TabFM matters. TabFM-Auto works by pairing the world knowledge and code generation of LLM coding agents (reading column names, units, domain formulas, and sentinels that synthetic-pretrained TFMs cannot see) with the strong, training-free numerical prediction of TFMs. Our ablations test each part of this pairing:

  • Removing world knowledge via schema anonymization cuts domain-task gains by 30% to 47% (General Response Table 2).
  • Removing TabFM so the same agent trains GBDTs, linear models, and neural networks (Table 2, p. 7) introduces retraining noise into CV feedback and drops performance by 510.8 Elo (0.4400 vs. 0.3371 G-Mean; 12 GBDT and neural agents also trail TabFM-Auto on MLE-Bench-Tabular in Figure 7).
  • Swapping TabFM for three other TFMs (TabICLv2, TabPFN-3, EXAONE-Tabular) improves all three by +88.7 to +143.3 Elo without re-search (Figure 5 & Table 7), showing that the gain comes from pairing coding agents with TFMs generally rather than TabFM alone.

Thank you for your replies to Q1 and Q2; both are resolved.

(W1, Memorization) The current answer does not do my raised issue justice. All TabArena datasets are public; benchmark and preprocessing results that improve on TabArena are public, and an LLM could pick these up and put them into pipeline.py, or it could use test labels from pretraining and write the optimal preprocessing based on the unseen data. The papers I referenced also raise this issue and show that LLMs (even back when the papers were released) have been pretrained on tabular datasets from the internet. In other words, contamination is not limited to LLMs predicting themselves. In its current form, this work still ignores this issue. It remains unclear whether this work would generalize to other settings because it lacks leakage-free investigations.

(W1, Inductive) Thank you for the great investigation. It shows that transductive learning indeed gave the model an unfair advantage. The Elo change is not insignificant. Are you planning to update the paper to use the "fair" numbers going forward?

(W2) The current reply, which points to the existing ablations, does not answer my issue. To make it more specific, some questions to put my previous statement into more context: does Claude, without the "setup" of TabFM-Auto, perform similarly? Why not use other public TFMs in an ensemble with TabFM? Why not enable TabFM with GBDTs?

Thank you, and we're glad Q1 and Q2 are resolved. And thank you again for thoughtful comments!


Memorization.

We agree that contamination can go beyond an LLM predicting labels. Our audit rules out memorized labels entering the predictions directly: no pipeline hardcodes targets or reads outside its workspace, and a change is kept only if it improves CV on fold 0's training rows. What we cannot rule out on public benchmarks is that the LLM's prior knowledge of these datasets shapes what it tries, and we will state this limitation explicitly. However, prior knowledge alone is not enough. The Table 2 agent is a generic MLE agent with the same LLM and data, but without TabFM or our TabFM instructions. It can reproduce any solution it remembers, yet it reaches only 1468.8 Elo, far below TabFM-Auto with the same LLM. This indicates that TabFM-Auto's gains are not explained by contamination alone.


Inductive numbers.

We consider transductive use legitimate. It uses only unlabeled test features, never labels, and matches batch prediction where the rows to be scored are known in advance, as on Kaggle and MLE-Bench. Since all TabArena baselines are inductive, however, we will rerun the search for the 18 of 306 dataset-configuration pairs whose pipelines were not inductive, this time with an inductive-only restriction in the agent's instructions. We will verify that every resulting pipeline is strictly inductive and report those numbers in the camera-ready. Until then, simply reverting those pipelines to zero-shot TabFM gives a conservative estimate: the best configuration reaches 1965.5 Elo (vs. 2013.0), and every configuration stays above every baseline, including TabFM+ (1856.0). Nonetheless, these fallback numbers can serve as the lower bound for TabFM-Auto-Inductive-Only.


W2.

1. Does a coding agent without the TabFM-Auto setup perform similarly? TabFM-Auto is already a plain coding agent: the unmodified CLI with a short instruction file, a reference guide to TabFM's settings and a starter pipeline, with no custom agent loop or search algorithm. Results do not depend on a particular harness: the six CLI–LLM combinations land within 75 Elo of each other. The table compares TabFM-Auto with the generic MLE agent of Table 2 (both Antigravity + Gemini 3.8 Flash):

TabFM-Auto agentGeneric MLE agent (Table 2)
Unmodified coding-agent CLI and LLM✓✓
Same data, dataset description and metric✓✓
Same budget (6 h), sandbox and CV✓✓
Can train GBDTs, neural networks or any other model✓ (TabFM stays the main predictor)✓
TabFM available✓✗
TabFM-specific instructions✓✗
TabFM reference guide✓✗
Starter file with four hooks around TabFM✓✗ (empty predict())
Plain TabFM scored first as a reference✓✗

The generic agent scores 510.8 Elo lower. Our instructions and guide mostly explain how to use TabFM, which any agent must be told when it is given a model it has never seen, so they come with providing TabFM rather than being extra engineering. What remains is the four-hook starter file and the rule that TabFM stays the main predictor, which together are the design of TabFM-Auto itself.

2. Why not combine TabFM with GBDTs? As the table shows, TabFM-Auto already allows this. The instructions only require TabFM to remain the main predictor. The agent may train GBDTs or other models and blend them in, and keeps a blend only if it improves cross validation loss. In the Claude Code + Opus 5 sweep, 11 of 51 final pipelines do so. For example, Amazon_employee_access is 50% TabFM, 24% logistic regression and 26% GBDT. To clarify our earlier statement that frozen TabFM makes all predictions: every prediction involves TabFM, which remains the main predictor, but some pipelines also blend in other models. We will make this clearer.

3. Why not ensemble TabFM with other public TFMs? This is a good direction and could be interesting for future research, yet we left it out deliberately in our project. Adding models to the agent's toolbox enlarges the search space and multiplies the cost of each evaluation, so fewer edits fit in the same budget. We kept one frozen model to isolate what pipeline search adds. Our transfer results suggest a cheaper route: the discovered pipelines, applied unchanged, also improve TabICLv2, TabPFN-3 and EXAONE-Tabular by 88.7 to 143.3 Elo (these runs include the blends above), so TFMs could be ensembled after the search without enlarging it. We leave this to future work.


Thanks again for discussing with us! Let us know if our response has resolved your concerns or if you have further questions.

Comments on the paper

1 comment

We thank all three reviewers for their thoughtful and constructive feedback. During the rebuttal period, we ran two new experiments pairing Antigravity with Gemini 4 Argon: (1) a sixth full 51-dataset TabArena sweep, and (2) a controlled schema-anonymization ablation. Below we summarize these new results and address the main points shared across reviews. Upon institutional approval, we will release our sandboxed evaluator, prompts, per-fold results, and all 306 discovered pipelines.

1. Overlap-free Fold-0 evaluation, G-Mean alongside Elo, and new Gemini 4 Argon sweep. Reviewers asked how TabFM-Auto performs on overlap-free Fold 0, why Fold-0 Elo shifts relative to All Splits, and whether gains hold across sweeps and strictly inductive pipelines. Because Elo measures pairwise wins against a reference pool, recomputing baseline Elo scores on 51 single splits instead of all 816 splits shifts the scale. Elo also counts narrow and wide error margins equally, and at high win rates a few single-split flips can move Elo noticeably. To give a direct measure of error reduction that does not depend on the baseline pool, we report 51-dataset Geometric Mean test error (G-Mean ↓\downarrow) alongside Elo:

MethodFold-0 G-Mean ↓\downarrowFold-0 GainAll-Splits G-Mean ↓\downarrowInductive G-Mean ↓\downarrowAll-Splits Elo ↑\uparrow
Antigravity, Gemini 4 Argon (new)0.3310+6.14%0.33600.33791969.5
Codex, Opus 50.3399+3.61%0.34000.34262013.0
Claude Code, Opus 50.3413+3.21%0.34020.34301993.6
Antigravity, Gemini 3.8 Flash0.3445+2.30%0.33710.34051979.6
TabFM+ (fixed wrapper, no search)0.3501+0.73%0.35130.35131856.0
TabFM (zero-shot baseline)0.35260.00%0.35500.35501785.3

On Fold 0 (zero overlap with pipeline search), Gemini 4 Argon has the lowest G-Mean error (0.3310, +6.14% over TabFM), slightly ahead of its All-Splits gain (+5.35%). All three frontier sweeps have nearly identical G-Mean on Fold 0 and All Splits. In addition, 49 of 51 Gemini 4 Argon pipelines (96.1%) are strictly inductive, and reverting the remaining 2 to TabFM keeps all six sweeps ahead of every baseline.

2. Why TabFM-Auto works and new schema-anonymization ablation. Reviewers also asked why pairing a coding agent with TabFM works and what each part contributes. The core intuition is simple: TabFM-Auto pairs the world knowledge and code generation of LLMs and coding agents with the strong, training-free predictive power of TabFM and other tabular foundation models (TFMs). Because TFMs are pretrained on synthetic numerical priors, they are strong tabular predictors in a single forward pass, yet they cannot read column names, units, domain formulas, or external files. LLMs have the opposite profile: they understand domain semantics and can write data pipelines, yet they are weak direct predictors on numerical tables, and retraining GBDTs or neural networks at every step makes cross-validation noisy.

To measure the role of world knowledge directly on Antigravity (Gemini 4 Argon), we replaced column names with generic identifiers (col_0, col_1, ...) and removed dataset names and descriptions under the same splits, budget, and frozen TabFM:

Dataset (Antigravity, Gemini 4 Argon)Full-Schema GainAnonymized-Schema GainGain Retained
diabetes+3.74%+2.60%69.5%
miami_housing+1.61%+0.97%60.1%
houses+1.58%+0.91%57.7%
diamonds+0.95%+0.50%52.7%
Mean (domain-formula tasks)+1.97%+1.24%60.0%

Removing column names and descriptions reduces the gain by 30% to 47% (40% on average) on tasks with domain formulas, while statistical cleaning, relational features, context sampling, and calibration retain the remaining 60%. On the other half of the pairing, removing TabFM so the same agent trains GBDTs and neural networks drops performance by 510.8 Elo (Table 2), while transferring our discovered Python hooks to three other frozen TFMs adds +88.7 to +143.3 Elo without re-search (Figure 5 & Table 7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.