acceptodds
Under review as a conference paper at ICLR 2027

TabEvo: LLM-Guided Evolution of Dataset-Conditional Programs for Tabular In-Context Learning

Abstract

A tabular in-context learner is a function from a labelled table and query rows to predictions, which tabular foundation models implement with a transformer pretrained on millions of synthetic tables. We show that the same function can be an explicit program found by large language model (LLM) guided search. TabEvo represents a learner as a meta-DAG: a directed graph whose edges carry code that fits a transformation or a predictor on the rows of the table at hand, whose nodes merge the resulting blocks, and whose head combines prediction blocks by cross-fitted non-negative stacking, so that no component reads a prediction made with its own label. An LLM agent loop grows a program one accepted edit at a time; an evolutionary search then edits it through parent-specific structured diffs, every child fitted on 15 development tables before entering a MAP-Elites archive. The resulting program has four exact Gaussian-process views of the input space, two residual learners, repeat-detecting gates and a gradient-boosted lane. Frozen and evaluated under a pre-registered protocol on 71 tables against 45 baseline configurations, it is within one point, by our equivalence rule, of six of the nine foundation models on 22 of the 26 fresh TabArena tables, 0.7 to 0.9 points behind TabFM and LimiX, and level with bagged CatBoost, RealMLP and a Super Learner of trained models, with no pretraining; on 24 fresh regression tables it is ahead of every trained model and of the Super Learner, about one point behind the newest foundation models. As a program it can be tested and edited: with two thousand mostly irrelevant features it keeps an R2 of 0.86 where the foundation models fall to 0.33–0.39, and an automated edit-and-test loop halved its graph and made it 6.5× faster at a 0.3-point cost in accuracy and no detected change in ROC-AUC.

Primary area:
generative models
Keywords:
tabular learning, tabular foundation models, in-context learning, LLM-guided evolution, evolutionary program synthesis, conditional computation graphs, synthetic data priors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DHy1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article introduces TabEvo, which represents table learning models as a graph. Each part of the graph can learn data changes or make predictions on new datasets. It also uses nested cross-fitting to pass predictions between parts and then combines them for the final output. The first program is built by a LLM, and then it's improved through evolutionary search based on scores on development data. The final model combines several Gaussian process models, a residual learner, a module for handling duplicates, and CatBoost. The authors tested TabEvo on 95 datasets and compared it with 45 benchmark settings. They also ran extra tests to study the different parts of the method, analyzed failure cases, and tested the function that automatically removes unnecessary parts of the program.

Critical strengths

1.TabEvo turns tabular learning into a program that can be changed and checked. It also clearly separates the steps used to train and test the model. 2.TabEvo performs well without pretraining on finished tasks. It also allows tests under different conditions and removes parts of the program that are not needed.

Critical weaknesses

1.TabEvo doesn't show how much the search method improves results beyond the strong starting program and manual changes. 2.TabEvo doesn't show a clear advantage over standard stacking when we consider failed runs, training costs, and similar Gaussian process models.

Questions for authors
  1. Compared to the first program and manual tweaks, what extra improvements can search find?
  2. When both use similar components, can a meta-DAG do better than regular stacking?
  3. Considering failed runs and training costs, does this advantage still hold for new tasks?
1 comment

Thank you for the careful reading and precise questions. We address each with results and paper references.

Q1. What does search add beyond the seed and manual changes?

The seed is itself produced by TabEvo's first stage: an LLM agent grows a minimal genome through accepted edits (Sec. 2.3). The separate construction experiment places the final program +2.22 points [0.84, 3.82] above a single exact GP with the same head on 15 development tables (Table 20).

Evolution then adds a direct CatBoost lane and its fitting rule. The final program shares the seed's head, nodes and other edges (Table 18). On 23 completed fresh regression tables, this addition yields +0.60 R2R^2 points [0.12, 1.44], with 20 wins, 2 ties and 1 loss. On all 24 tables, failures scored as worst, the gain remains +0.58 [0.12, 1.40]; both Holm p<.001p < .001 (Table 60). Fresh classification accuracy changes by +0.06 [−0.05, +0.17] (Sec. 4).

The manual intervention restored the seed's non-negative head after a ridge-headed candidate transferred poorly; the CatBoost lane came from evolution. These results come from one evolutionary run with 51 evaluated children (Sec. 2.3).

Q2. Can a meta-DAG outperform stacking with similar components?

Stack-TD combines tuned CatBoost, LightGBM, and RealMLP using the same stacking head as TabEvo. On fresh tables both methods completed, TabEvo was within on regression led by +0.64 points [+0.09, +1.26]. Adding an exact ARD GP to the stack narrowed the gap to +0.24 points [−0.28, +0.63] on 17 common completed tables. The GP alone trailed TabEvo by 3.54 points and lost on all 23 paired tables. Stack-TD + GP uses similar components, but it is not a flat stack of every TabEvo block (Tables 1 and 60).

The advantage was clearer in the controlled noise-feature experiment. With 500 training rows, 2,000 features (only five informative), and three data seeds, TabEvo retained R2=0.86R^2 = 0.86, versus 0.75 for Stack-TD, 0.67 for CatBoost, and 0.33–0.39 for tabular foundation models (§5.1).

In short, TabEvo was close to a strong stack on real tables and retained more predictive accuracy when most features were noise

Q3. Do gains survive failures and training costs on new tasks?

Regression quality gains survive failure accounting. Across all 24 fresh regression tables, TabEvo leads bagged CatBoost by +1.36 [0.29, 2.47], Holm p=.028p = .028, and bagged LightGBM by +1.37 [0.52, 2.21], Holm p=.001p = .001 (Table 60).

Against Stack-TD + GP, the failure-inclusive margin is +2.14 [0.21, 4.79], Holm p=.035p = .035: TabEvo completes 23/24 tables versus 18/24 for the stack. This reflects better completion alongside comparable quality on common completed tables. Against plain Stack-TD, +0.40 [−0.31, +1.10] does not meet the pre-registered practical-advantage criterion. On classification, failures give Stack-TD a 1.06-point lead [0.09, 2.56] (Table 59).

The full program is expensive: median fit+predict is 646 s over 46 completed TabArena fits on an H100 slot, versus 20 s for TabICLv2 and 21 s for bagged CatBoost on 8 CPU threads (Table 62; different completed-table sets). The evolutionary run used an estimated 46 GPU-hours and 18.6 M LLM tokens, excluding seed development but the frozen graph is then reused without rerunning search (App. D.6).

Program editability provides a measured cost–quality trade-off. The profile–edit–test loop cuts 23 edges to 12 and achieves 6.5×6.5\times speed-up on development tables, at score 0.7894 versus 0.7896. On fresh classification, the pruned program completes 23/26 tables, takes a median 49 s, and loses 0.3 accuracy points with no detected change in ROC-AUC or log-loss (Sec. 5.3, App. F.5).

Without pretraining, TabEvo is within one point of seven of nine foundation models on completed fresh classification tables and level with the strongest trained models. On 23 fresh regression tables it averages R2=0.713R^2 = 0.713, ahead of individual trained models on average and ~one point behind the newest foundation models. Its interpretability is that of an explicit, inspectable and editable program: gates explain repeated-row behavior (Sec. 5.1), blocks can be ablated (Table 2), and edit-and-test halved the graph without changing development score. We hope this addresses your questions.

Reviewer_81VtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TabEvo, which represents a tabular in-context learner as a meta-DAG shared across datasets, and uses LLM-guided structured program evolution to search for the computational structure within it. The program consists of multiple GP, warping, metric learning, CatBoost, MLP, and repeat-detection components, and is combined through cross-fitted non-negative stacking. Upon evaluation, TabEvo approaches most tabular foundation models on classification, and is able to surpass multiple trained baselines on regression, but its advantages relative to the Super Learner and strong foundation models are relatively limited.

Critical strengths
  1. It proposes and empirically validates a non-neural-network implementation path for a tabular in-context learner. This point is extremely useful, as it makes the transfer learning rule explicit and executable, which in many practical deployment situations has many advantages compared to neural networks. This greatly aroused my interest and I consider this method to indeed be very important.
  2. The experimental scope is very broad, and the transparency is very high. The experiments are very rich, and the disclosed content is also extensive, which increases the credibility of the experiments.
Critical weaknesses

The core contribution of "LLM-guided program evolution" is not supported by the evidence. The performance is not very good, and the incremental contribution of the evolutionary search is very limited. The final program's improvement over the initial one on new tables is very small. Moreover, the paper as a whole involves human intervention, which does not match the title, and to a large extent weakens my confidence in the core contribution.

Questions for authors
  1. I hope the authors can clarify what the core contribution of this paper actually is: is it "LLM-guided program evolution," or a carefully designed system? The authors need to provide a clearer explanation of their own growth.
  2. Why is human intervention necessary? What is the effect of removing the intervention? Are there other intervention methods for this step? Which intervention is better? Can this step be omitted, eliminating the need for human labor?
  3. Must the method rely on a strong LLM? Can it be reproduced with open-source models?
1 comment

On the small gain

We thank the reviewer for the observation regarding the incremental performance gain on new tables. Given that the seed prior is already strong, a small gain is the expected consequence, and we would prefer not to frame this as a failure of the evolutionary search.

While the headline metric shows only a modest difference, we believe the broader value of TabEvo lies in making the combinatorial design space tractable to explore. This enables the discovery of programs that are interpretable, inspectable, and editable. While scalar baselines remain essential, we hope this structural capability serves as a useful contribution. We have revised the paper to better highlight this aspect of our work.

Q1. Is it "LLM-guided program evolution," or a carefully designed system?

We sincerely thank the reviewer for this question! To be clear, the core contribution of this paper is the carefully designed system—specifically, the meta-DAG representation for tabular in-context learning. We build upon the LLM-guided evolutionary machinery provided by GigaEvo, which implements MAP-Elites, asynchronous evaluation, and LLM mutation. However, our contribution is the structural design that makes the machinery applicable to tabular tasks: the meta-DAG representation, cross-fitting semantics,evaluator and parent-specific structured diffs.

The empirical evidence shows that this structured DAG interface makes the search reliable and effective:

  • High executability: Only 5/59 proposals failed to parse, and just 1/51 evaluated children failed during fitting.
  • Strong evolutionary signal: 43/50 valid children outperformed the seed on development fitness, and 29/50 improved their own parent.

To test whether the structured DAG interface is necessary, we compared it against a standard evolutionary search over unrestricted Python code. As detailed in Appendix J, this unrestricted baseline evaluated approximately 3,700 candidates over ~530 evaluation-hours, yet it demonstrated worse transferability than TabEvo.

Ultimately, LLM-guided evolution is the engine, but the meta-DAG representation is the core contribution -it is what makes the evolutionary search tractable, supports reliable execution and transfer to tabular tasks.

We will revise the paper to explicitly foreground this distinction.

Q2. Why is human intervention necessary? What is the effect of removing the intervention? Are there other intervention methods for this step? Which intervention is better? Can this step be omitted, eliminating the need for human labor?

We thank the reviewer for these precise questions. We address them directly:

We used human intervention as a pragmatic design-space constraint. Program synthesis has a huge combinatorial search space, and without structural constraints the search wastes compute on degenerate configurations (e.g., the ridge-headed genome). Manual pruning guided the search toward high-value regions and saved evaluation compute, but the framework does not depend on this intervention.

The step can be omitted. LLM-generated Finalist A autonomously discovered the non-negative head and achieved equivalent development fitness (0.790 vs. 0.789) without human labor. Finalist B was selected only because of our predetermined selection rule. Thus, an autonomous evolutionary search can discover the necessary structure on its own.

Two automated alternatives exist: (1) prompt-level constraints that forbid specific patterns before code generation, and (2) static validation using AST parsing to reject degenerate proposals before evaluation. Prompt-level constraints are lightweight and cheap but less strict; static validation is robust but requires engineering rule-based checks. Manual intervention was used in the current study to save compute. For full autonomy, automated alternatives are the most promising direction.

Q3. Is a strong LLM necessary; can an open model be used?

The evolutionary engine is model-agnostic, though search quality remains limited by the model's ability to reason over iterative feedback. GigaEvo has successfully used open models, including Qwen3-235B-A22B-Thinking and Llama 3, demonstrating that open models can technically drive the evolution.

However, model capability plays a significant role in the structured DAG setting. In a quick post-submission TabEvo test, a smaller open model (Qwen3-4B-Instruct-2507) repeatedly violated the structured schema and failed to produce an evaluated child. We therefore do not claim that GPT-5.6 is strictly necessary, but our evidence suggests that larger, reasoning-capable open models can effectively drive GigaEvo searches.

Reviewer_brDpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The following paper studies the task of learning a program using a language model for tabular output prediction tasks instead of prompting language models to predict outputs. The authors propose a technique that uses a Directed Acyclic Graph (DAG) representation to evolve the program. The programs contain nodes that merge intermediate representations or predictions, and edges that contain Python code for fitting a transformation or predictor.

Critical strengths

The paper covers an interesting idea of programs with gaussian operators towards predicting table outputs.

Critical weaknesses

The paper reports the held-out set as part of the evaluation criteria and measures performance of the technique and baselines. The held out tables were used to select the final program and to inform methodological decisions during search. In particular the authors also intervene with the automated process in the following ways - after observing poor transfer of the ridge-headed genome on the held-out set, the authors restricted parents to the non-negative head. Additionally, the authors also manually introduced a head replacement into the selected lineage. Finally, the final program cannot be attributed solely to the proposed autonomous LLM-guided search procedure. Since the paper studies a single program it is difficult to characterize if the technique generally works towards writing programs in an automated manner for table output generation.

Questions for authors
  • The authors note that evolution only helps a little? Is it because the LLM-generated seed was good? Why isn't evolution contributing to learning?
  • why do we need LLM-guided evolutionary program synthesis if a manually specified Super Learner with GP + boosted trees + neural network gets essentially the same result?
  • What is the performance of base gpt-5.6 on the table output prediction tasks ?
2 comments

We thank the reviewer for the critical assessment. We take concerns seriously and address each point directly.

On the “held-out” terminology and evaluation protocol

Appendix A specifies the role of these ten tables. The label “held-out” denoted only their exclusion from search fitness; they served as validation data to select between two finalist programs. To make the role of each set clearer, the revision will consistently refer to these as the validation set and the fresh tables as the independent test set.

On the interventions

We acknowledge these interventions transparently. They are not ad-hoc fixes, nor do they undermine autonomous search; they are pragmatic design-space constraints. Program synthesis operates in a massive combinatorial space. Without structural constraints, the search wastes compute on degenerate configurations (e.g., the ridge-headed genome). We will explicitly distinguish these meta-level decisions from autonomous LLM mutations in the revision, discussing why such priors are essential for cost-effective program synthesis.

On the concern that the paper studies a single program

A core advantage of our approach is synthesizing a single, frozen program via structured LLM-guided evolution, rather than a diverse program population. The 50 fresh test datasets evaluate how well this program transfers to new tables. Measuring how reliably the search rediscovers such a program across independent runs is a distinct research question, outside the current scope.

Q1. The authors note that evolution only helps a little? Is it because the LLM-generated seed was good? Why isn't evolution contributing to learning?

We disagree that a small gain implies evolution isn't contributing to learning; evolution produced a concrete modification: adding a CatBoost lane to the seed's GP and residual lanes (Appendix C.2). The seed had limited headroom, making a small gain expected, not a failure.

Moreover, we would prefer not to reduce TabEvo’s contribution to a scalar gain. TabEvo makes the design space tractable, yielding interpretable, inspectable, and editable programs. The contribution is a legible design space, not merely a larger R2R^2.

Q2. why do we need LLM-guided evolutionary program synthesis if a manually specified Super Learner with GP + boosted trees + neural network gets essentially the same result?

Rather than framing this as "essentially the same result," we view the manual stack as a highly competitive baseline that nonetheless differs from TabEvo in several important respects. On fresh regression tables, TabEvo achieved the highest mean R2R^2: 72.90 vs. 72.66 (Stack-TD + GP) and 72.09 (Stack-TD). Under a one-hour budget, TabEvo predicted 23/24 tables vs. 18/24 for Stack-TD + GP. Scoring incomplete runs as worst-performing, TabEvo leads by 2.14 points across all 24 tables (95% CI [0.21, 4.79], p=0.035).

Additionally, TabEvo’s graph enabling qualitative insights: which connections are exploited, and which design choices matter. A manual stack fixes topology in advance and cannot answer these mechanistic questions without re-engineering for each hypothesis.

Q3. What is the performance of base gpt-5.6 on the table output prediction tasks ?

We appreciate the reviewer's suggestion about this baseline. However, we believe that direct prompting is not the most suitable fit for tabular prediction. LLMs are trained on sequences; they are not trained to reason over relational or tabular data structures. Serializing a table into text discards the very structure - column semantics and feature interactions - that tabular methods are designed to exploit. Moreover, the approach is prohibitively expensive: every batch of test rows requires reloading the entire labeled context, consuming millions of tokens per table with no reuse.

Nevertheless, to address this question empirically, we evaluated this baseline on the 5 fresh test datasets that fit within practical context limits and matched our evaluation protocol, allowing a direct comparison under the same conditions.

DatasetGPT-5.6TabEvoGPT-5.6 − TabEvo
credit-g67.37%76.3%−8.93 pp
fitness-club76.20%76.4%−0.20 pp
is-this-a-good-customer80.35%88.5%−8.15 pp
seismic-bumps92.11%93.4%−1.29 pp
marketing-campaign81.93%89.0%−7.07 pp

Base GPT-5.6 is worse than TabEvo on all datasets, with an average gap of approximately 5.1 percentage points.

These results confirm that TabEvo’s advantage arises from its synthesized graph structure and evolutionary search, rather than the raw predictive capabilities of a base LLM.

I thank the authors for their response and additional evaluation. A few follow-up questions regarding the comparison against GPT-5.6

  • Was an agentic harness employed here? What do the authors mean when they say similar conditions for evalutation?
  • What is the failure mode for GPT-5.6 when compared with TabEvo?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.