TabEvo: LLM-Guided Evolution of Dataset-Conditional Programs for Tabular In-Context Learning

A tabular in-context learner is a function $f(D,X^{\ast })$ from a labelled table and query rows to predictions, which tabular foundation models implement with a transformer pretrained on millions of synthetic tables. We show that the same function can be an explicit program found by large language model (LLM) guided search. TabEvo represents a learner as a meta-DAG: a directed graph whose edges carry code that fits a transformation or a predictor on the rows of the table at hand, whose nodes merge the resulting blocks, and whose head combines prediction blocks by cross-fitted non-negative stacking, so that no component reads a prediction made with its own label. An LLM agent loop grows a program one accepted edit at a time; an evolutionary search then edits it through parent-specific structured diffs, every child fitted on 15 development tables before entering a MAP-Elites archive. The resulting program has four exact Gaussian-process views of the input space, two residual learners, repeat-detecting gates and a gradient-boosted lane. Frozen and evaluated under a pre-registered protocol on 71 tables against 45 baseline configurations, it is within one point, by our equivalence rule, of six of the nine foundation models on 22 of the 26 fresh TabArena tables, 0.7 to 0.9 points behind TabFM and LimiX, and level with bagged CatBoost, RealMLP and a Super Learner of trained models, with no pretraining; on 24 fresh regression tables it is ahead of every trained model and of the Super Learner, about one point behind the newest foundation models. As a program it can be tested and edited: with two thousand mostly irrelevant features it keeps an R2 of 0.86 where the foundation models fall to 0.33–0.39, and an automated edit-and-test loop halved its graph and made it 6.5× faster at a 0.3-point cost in accuracy and no detected change in ROC-AUC.