acceptodds
Under review as a conference paper at ICLR 2027

Agent-Designed Search Spaces for Tabular Machine Learning

11 views

Abstract

Despite the rapid progress of LLM-based agents for planning, code generation, and debugging, their practical value for tabular machine learning remains underexplored. In this paper, we investigate a concrete use case: whether state-of-the-art agentic AI systems can design extended HPO search spaces for established tabular models that outperform the standard published search spaces of these models. Specifically, we represent each tabular model as a modular pipeline covering preprocessing, embeddings, architecture, training, and inference. We then task the agent to propose candidate code implementations for each module and use a classical HPO algorithm to jointly optimize over these candidates and the model's default hyperparameters. Compared with the base HPO spaces, the expanded search spaces improve four of the five model families across a suite of 45 datasets, with average relative gains of 0.6%, rising to 2.0% on small-to-medium regression datasets. Notably, these gains are obtained at matched budgets: the enlarged spaces outperform the base under the same number of tuning trials and ensemble members. The gains transfer to the recent TabArena benchmark, where the tuned and ensembled agentic spaces improve the Elo of four of the five model families and the two strongest agentic ensembles surpass AutoGluon's ensemble of conventional models. Under matched budgets, neither an end-to-end agent loop nor a human-curated bank of AutoML-library components reproduces these gains, suggesting that the value of LLM agents lies in expanding the design space. The code, prompts, and generated search spaces will be released upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tabular, deep learning, agents, hyperparameter optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aAfyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLM agents can improve tabular machine learning by expanding the hyperparameter optimization (HPO) search spaces of existing models. The authors decompose each model into modular components, such as preprocessing, embeddings, architecture, training, and inference. An LLM agent proposes and implements alternative methods for these components, which are then jointly optimized with the original hyperparameters using classical HPO.

The approach is evaluated on five tabular model families across 45 datasets, with additional experiments on TabArena. The results show improvements over the original search spaces, particularly on small-to-medium regression tasks and in ensemble settings. The paper also examines agent generation stability, candidate pool size, and comparisons with existing AutoML approaches.

Critical strengths
  1. The paper addresses a practical problem in tabular machine learning. Separating agent-based candidate generation from classical HPO is a sensible design, as it allows the generated search space to be reused across datasets without repeatedly invoking the agent.

  2. The experimental evaluation is relatively comprehensive, covering multiple model families, datasets, and evaluation protocols. The authors also provide useful analyses of candidate diversity, search-space size, and stability across different agents and independent generations. These experiments strengthen the empirical findings.

Critical weaknesses
  1. Insufficient discussion and comparison with prior agent-based AutoML methods. The paper does not adequately position its contribution relative to existing LLM-agent-based AutoML approaches, such as SELA [1], AutoML-Agent [2], and AIDE [3]. In particular, SELA already explores using LLM-generated suggestions to construct search spaces across different ML pipeline stages, followed by search-based optimization. While the proposed method differs by generating reusable, model-specific modules and delegating optimization to classical HPO, these distinctions deserve a more thorough discussion. Moreover, the experimental comparisons focus primarily on traditional AutoML systems, while the only agent-based baseline is an end-to-end autoresearch loop evaluated on four datasets. Although directly comparing these frameworks may require adapting their settings, a more comprehensive discussion and comparison with existing agent-based approaches would help clarify the contributions and practical advantages of the proposed method.

  2. Limited isolation of the sources of performance improvements. The comparison with the human-curated AutoPyTorch component bank is informative, but the two search spaces differ substantially in size and coverage. For example, the human-curated bank lacks numerical embedding and target preprocessing alternatives included in the agent-generated space. Since Section 5.3 also shows that performance improves as the candidate pool grows, it remains unclear how much of the improvement comes from the effectiveness of agent-generated candidates rather than the broader coverage of existing techniques. A more closely matched comparison would help clarify this. Additionally, the TabICLv2 baseline is evaluated without HPO, whereas its agentic variant receives 100 tuning trials. This makes it difficult to distinguish the benefits of search-space expansion from those of additional tuning.

[1] Chi, Y., et al. (2024). SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning. arXiv:2410.17238.

[2] Trirat, P., Jeong, W., & Hwang, S. J. (2025). AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. ICML 2025. Paper.

[3] Jiang, Z., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.

Questions for authors
  1. Could the authors further clarify how the proposed framework differs from existing agent-based AutoML approaches? In particular, what do the authors consider the main advantage of generating reusable search spaces over existing agent-driven pipeline optimization methods?

  2. Section 5.3 shows that performance generally improves as the candidate pool grows. To what extent do the authors believe the improvements come from the quality of agent-generated candidates, rather than simply having a larger and more diverse search space? Are there any observations from the experiments that could help distinguish these factors?

  3. For TabICLv2, the original model is evaluated without HPO, while the agentic variant receives additional tuning trials. Could the authors clarify how much of the observed improvement is attributable to the proposed search-space extensions, as opposed to the benefit of tuning itself?

5 comments

Thank you for the detailed review and the clear questions.

Q1: Difference from agent-based AutoML (SELA, AutoML-Agent, AIDE).

We will add a discussion of these works. They solve a broader task: given a dataset and a prompt, build a full pipeline for that dataset. Our task is narrower: given a model implementation, agent writes a search space that improves it, once and offline. On a new dataset, the user runs standard HPO over the released modules.

SELA, AIDE, AutoML-AgentTabFM-Auto [1] (concurrent)Ours
Agent runsper datasetper datasetper model
Agent API cost-$17.6K in total (≈\approx$70 per dataset)≈\approx$30 per model
Agent sees the datayesyesno
LLM needed on a new datasetyesyesno
Agent proposeswhole pipeline including modeldata pipelinemodules inside a given model

Q2. Main advantage of our method.

Our method is particularly attractive for engineers and researchers working on tabular data, because:

  1. Per-dataset agents read the data. Public benchmark datasets are likely in the LLM's training data, so leakage cannot be ruled out. In our initial experiments with autoresearch, we found that an agent can recognize datasets from column values. There are numerous papers such as RealMLP, TabM, Revisiting Deep Learning [2] that disclose optimal architectures and hyperparameters per dataset, and LLM agents can use them. Since LLMs can lie even in their thinking process, there are no good guardrails to prevent this.
  2. Privacy. In many applications, data or metadata cannot be shared with LLM.
  3. High cost. Per-dataset method are costly, especially given the current API prices (our autoresearch baseline costs $20-50 per dataset and concurrent work of TabFM-Auto reports $70 per dataset). In our case, the search space is generated once per model and re-used without LLM.
  4. Modularity and practicality. Since our output is a fixed search space, it works with existing tools and does not need an LLM at deployment. We show that our search spaces are plug-and-play by adding our spaces to AutoGluon.
  5. State-of-the-art results. In time for this review, we evaluated TabM-A on the widely recognized TabArena (full benchmark, 51 datasets). It reaches the highest Elo among non-foundation models. It is also, up to this date, the first agentic (designed by an agent) models on that benchmark.
Tuned + ensembledElo [95% CI]TabM-A lower error onWilcoxon p
TabM-A (ours)1514 [1485, 1547]--
TabPFN-3 (foundation model)1631 [1583, 1688]11/51-
AutoGluon best_quality1484 [1443, 1525]32/51-
RealMLP1478 [1437, 1517]30/51-
TabM1425 [1390, 1469]41/51< 0.001

Q3: Quality and quantity of the agent-generated candidates and comparison with human-curated banks

We see the size and coverage of the space as the benefits that an LLM agent can provide over traditional (human-engineered) AutoML systems. The practioners are untied from package (AutoKeras, AutoPyTorch) releases. For example, a re-generation of the TabM modules with the newer Opus 5.5 added the Muon optimizer and arithmetic feature interactions, and improves on TabM-A on TabArena-Lite (Elo 1514 vs. 1474, general reply).

AutoKerasAutoPyTorchAgentic MLP$^\dagger$
Components (Table 12)44461 / 57 (reg. / clf.)
Module axes covered3610 / 9
Last release3.0.0, Nov 20250.2.1, Aug 2022 (no commits since)on demand ($\approx$25-100 min)

Two observations speak to the quality of the candidates. First, no single candidate carries the gain: within the most important modules, no candidate wins on more than 12 of the 45 datasets, and every candidate wins on at least one (Figure 3). Second, the quality is reproducible: ten independent re-generations by two agents (Claude Code and Codex) all improve over the base MLP$^\dagger$ space on all four aggregate metrics (Table 4).

Q4: TabICLv2 tuning.

We tuned base TabICLv2 over TabArena's own search space (TabArena-Lite, 51 datasets).

EloTabICLv2TabICLv2-AΔ\Delta
Default156115610
Tuned16001616+16
Tuned + ensembled16231632+9

Thank you for suggesting this ablation. With a matched budget, TabICLv2-A still scores higher after tuning and after tuning and ensembling (28 of 51), but the gap is narrower. We will add this comparison to the revision and adjust the TabICLv2 wording in the text accordingly.

References

[1] Fu et al. TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models. arXiv:2609.37989, 2026.

[2] Gorishniy et al. Revisiting Deep Learning Models for Tabular Data. NeurIPS 2021.

Thank you for the response and for providing the additional comparison with TabICLv2.

However, I still have questions regarding the comparison with existing AutoML component libraries. I notice that most of the components proposed by the agent are general-purpose; how does this differ from simply writing additional plug-ins for existing AutoML systems? Furthermore, some of the baselines do not use the latest versions—for instance, AutoKeras 1.1.0 is used instead of 3.0.0. While there may be a valid reason for this choice, it is not explained in the paper.

I would also like to gain a deeper understanding of why this approach is effective. While the cross-dataset results demonstrate the utility of an expanded search space, it is perhaps not surprising that a set of general-purpose components can be applied across different datasets. Are there any specific findings that highlight what makes the components or combinations proposed by the agent unique? Otherwise, it seems plausible that similar gains could be achieved by simply incorporating such components into other AutoML systems. Given the differences in scope, the current comparison with manually curated component libraries does not yet fully address this point.

Consequently, I remain reserved regarding the paper’s contributions and will maintain my current score.

Thank you for the follow-up. Your argument, as we understand it, is: (1) the agent's components are general-purpose, hence (2) they could be added to any AutoML system as plug-ins, hence (3) the gain is not specific to our method. We address each step.

1. Many components are not general-purpose

The agent writes them for a given model, after reading its code.

ModelExamples of generated candidates
Any neural modelMuon, Lion, SAM, and Lookahead optimizers, SWA, self-distillation
TabMlow-rank (LoRA-style) per-member adapters on the shared weights, a diversity loss between the ensemble members, variance-weighted aggregation of the member predictions
RealMLPBatchEnsemble layers in place of its independent ensemble members, shared categorical embedding tables with per-member adapters, residual blocks with RealMLP's data-dependent initialization
TabICLv2feature-permutation schemes and aggregation rules for its in-context ensemble

2. Hence they cannot simply be plugged into any AutoML system

A per-member adapter needs TabM's shared-weight ensemble, and a data-dependent initialization needs RealMLP's training procedure. AutoKeras and AutoPyTorch search over their own backbones, so these candidates have nothing to attach to. The unit that transfers is the model together with its space. Where a system contains the model, the transfer works, and we show it: AutoGluon (+A) receives our agentic RealMLP and TabM$^\dagger$ spaces and improves on AutoGluon on all four aggregate metrics (Figure 2).

3. Why the approach is effective

The method works because three facts hold together.

  • Tabular tasks are heterogeneous. TabArena contains biological, financial, real-estate, and engineering datasets, and each kind favors different components. For example, engineering targets often depend on ratios and products of the input features, and real-estate prices depend on location, for which piecewise-linear embeddings are effective.
  • Hence the gain comes from per-dataset selection, not from one component. No candidate wins on more than 12 of the 45 datasets, and every candidate wins on at least one (Figure 3).
  • The agent supplies many diverse candidates. In our experiments, a larger pool of candidates generally helps. The agent also follows recent literature: the Opus 5.5 generation added the Muon optimizer and pairwise arithmetic feature interactions, both from 2025-2026 papers, and it improves on TabM-A on TabArena-Lite (Elo 1514 vs. 1474).
  • Standard HPO finds the right combination for each dataset. Unlike per-dataset agent loops, the search over the agent's candidates needs no LLM.

As a result, a single model outperforms a full AutoML system. On the full TabArena, TabM-A has a lower error than AutoGluon best_quality on 32/51 datasets and the highest Elo among non-foundation models.

AutoKeras version

Thank you for noticing this. We ran AutoKeras 1.1.0. Version 2.0.0 removed the tabular tasks, and we overlooked that 3.0.0 restored them. We have now compared the two releases. The tabular search space is the same in both: the architecture, the optimizer choices, and the tuner with its initial configurations. The implementation differs in three places:

1.1.03.0.0
Categorical featuresIDs assigned by frequencyIDs assigned in sorted order
TrainingAdamW with warmupAdamW without warmup
Model used for predictionfinal refitted modelbest checkpoint of the search

We will rerun the AutoKeras baseline with 3.0.0 and report it in the paper.

Question

AutoGluon (+A) is the comparison that you describe for a system that contains our models. Could you tell us which additional comparison would address the remaining difference in scope?

Reviewer_BhxWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses LLM coding agents (specifically, Claude Code and Codex) to expand the hyperparameter search space of tabular models: the agent implements alternative pipeline modules once per model family, and a standard HPO sampler then searches the enlarged space on each dataset. The expanded spaces improve four of five model families on 45 datasets and TabArena.

Critical strengths
  1. The approach has practical engineering value: it uses an agent to expand the search space of existing tabular models with implemented modules, which is otherwise tedious to do by hand.
  2. The experimental coverage is broad, spanning five model families and two benchmarks (45 curated datasets and TabArena).
Critical weaknesses

My major concern is that the paper offers a solid engineering improvement, but the gains are modest. In Figure 2 and the per-dataset results of Table 22, the improvement of Agentic HPO over the base space in the Individual setting is small, and for several datasets, the agentic space is worse. The only bootstrap CIs reported (Table 14) show significance for MLP† but not for TabM† or RealMLP, and none are given for LightGBM or TabICLv2. In the Ensemble setting, AutoGluon remains the strongest overall.

Besides, my understanding is that the agent generates the search space without using target-dataset information, allowing the same space to be reused across datasets. And this is the stated advantage of the method. However, I think a comparison with search spaces generated using both the model and training-data characteristics would help: if dataset conditioning offers little or no additional improvement, this would strengthen the case for the proposed reusable approach. Finally, for some minor issues with presentation, the method is referred to inconsistently as "agentic space", "Agentic", and "Agentic HPO", and Figure 5 is not legible.

Questions for authors

Please see Weaknesses.

4 comments

Thank you for your review and for the suggestions on presentation.

Q1. Size and significance of the gains.

In Figure 2, we report 95% CIs across datasets for every method. Following your comment, we add paired comparisons of each agentic space with its own base space on all 45 datasets: mean relative improvement (%), dataset-bootstrap 95% CI, and Wilcoxon signed-rank p-value. Table 14 covers only the neural families because the human bank consists of network components (backbones, initializers, optimizers, schedulers) which do not apply to LightGBM or TabICLv2.

FamilyTunedTuned + ensembled
MLP$^\dagger$+1.28 [+0.40, +2.41], p = 0.011+1.33 [+0.55, +2.35], p < 0.001
TabM$^\dagger$+0.40 [-0.33, +1.27], p = 0.42+1.08 [+0.30, +2.07], p = 0.007
LightGBM+0.82 [+0.08, +1.77], p = 0.11+0.22 [-0.78, +1.21], p = 0.70
RealMLP-0.21 [-0.74, +0.30], p = 0.47+0.33 [-0.12, +0.84], p = 0.16
TabICLv2+0.54 [+0.05, +1.17], p = 0.13-

As in Table 2, the gains are largest on small-to-medium regression (tuned single models, 14 datasets):

FamilyΔ\Delta over base spaceWinsp
MLP$^\dagger$+2.8%11/140.003
LightGBM+2.0%11/140.025
TabM$^\dagger$+1.7%10/140.14

For scale, moving from the tuned MLP$^\dagger$ to the tuned TabM$^\dagger$, a separately published architecture, gains +1.9% on the same 45 datasets. The agentic space gives MLP$^\dagger$ +1.3% (tuned and ensembled) without a new architecture or any change by hand.

The strongest evidence of the strength of our method comes from an independent protocol. In time for this review, we evaluated TabM-A on the full TabArena (51 datasets, 816 splits). It reaches the highest Elo among non-foundation models:

Tuned + ensembledElo95% CI
TabM-A (ours)1514[1485, 1547]
AutoGluon best_quality1484[1443, 1525]
RealMLP1478[1437, 1517]
TabM1425[1390, 1469]
LightGBM1408[1381, 1433]
CatBoost1399[1365, 1431]

Head to head, TabM-A has a lower error than RealMLP on 30/51 datasets, than AutoGluon on 32/51, and than the official TabM on 41/51 (Wilcoxon p < 0.001). A re-generation with the newer Opus 5.5 already improves on TabM-A on TabArena-Lite (Elo 1514 vs. 1474, general reply).

Q2. AutoGluon in the ensemble setting.

AutoGluon's best_quality preset bags, stacks, and ensembles 110 configurations from 7 model families, with a portfolio meta-learned offline (TabRepo [1]), and is trained on train + validation. Each of our ensembles uses a single family and is trained on train portion only.

Additionally, AutoGluon can be viewed as complementary to our work. We report the results for AutoGluon (+A), which receives our agentic RealMLP and TabM$^\dagger$ spaces and improves on AutoGluon on all four aggregate metrics (Figure 2).

Finally, on full TabArena (see the new results above), the single-family TabM-A has a lower error than AutoGluon best_quality on 32/51 datasets.

Q3. Dataset-conditioned spaces.

We agree that the comparison is interesting, but it is hard to separate memorization from genuine skill in LLM agents. In our experiments with autoresearch, we have tried exactly this and found that an agent can recognize datasets by column values. Tuned configurations and per-dataset results for them are already published: TabArena datasets and splits appeared in RealMLP and OpenML datasets in TabM. Monitoring the agent's reasoning does not settle this because LLMs can lie in their chain-of-thought [3, 4].

Further, running per dataset is associated with much higher costs: we spent around $10-30 per model, <$150 for the whole study. For comparison, the concurrent TabFM-Auto [2], which runs its agent per dataset, reports $17.6K in API fees on TabArena (≈\approx$70 per dataset per model). We publish our expanded HP sets, so practitioners can deploy them for free without LLM. We will expand the discussion of dataset-specific generation in the Limitations.

Dataset-conditioned generationOurs
Agent sees the datayesno
Risk of memorized public datasetsyesno
API cost for our 65 distinct datasets≈\approx$2,000 per model family≈\approx$30 per model family
Reusable on new datanoyes

Q4. Presentation.

We will use one name throughout: "agentic search space" for the method and the suffix "-A" for the resulting models (e.g., TabM-A). We will replace Figure 5 with a compact version that shows only the model families under study and the reference systems.

References

[1] Salinas and Erickson. TabRepo. AutoML Conference 2024.

[2] Fu et al. TabFM-Auto. arXiv:2609.37989, 2026.

[3] Turpin et al. Language Models Don't Always Say What They Think. NeurIPS 2023.

[4] Chen et al. Reasoning Models Don't Always Say What They Think. arXiv:2505.05410, 2025.

Dear Reviewer BhxW,

Thank you again for your review. As the discussion period is coming to an end, we would like to ask whether our reply has addressed your concerns. In brief:

  • Size and significance of the gains (Q1): we added paired bootstrap CIs and Wilcoxon tests for all families, and the full TabArena evaluation, where TabM-A has the highest Elo among non-foundation models.
  • AutoGluon (Q2): on the full TabArena, the single-family TabM-A has a lower error than AutoGluon best_quality on 32/51 datasets.
  • Dataset-conditioned spaces (Q3) and presentation (Q4): we explained the leakage and cost issues, unified the naming, and replaced Figure 5.

If any point remains open, we would be glad to answer it in the remaining time.

Thank you so much for the rebuttal. I appreciate the authors’ response to my concerns. Regarding Q3, I agree that the cost consideration and data contamination issue in the agent are reasonable. However, I still think that at least a small-scale experiment would probably be helpful to demonstrate that the dataset-agnostic agentic search space achieves comparable gains. Overall, I will keep my current score.

Thank you for the follow-up. We ran the small-scale experiment that you suggested.

Setup

For TabM$^\dagger$ and Opus 5.5, we compare two ways of generating the search space on four datasets (three of our 45 and MIC from TabArena):

  • Dataset-agnostic (ours): one generation without any data, reused on all datasets.
  • Dataset-conditioned: one generation per dataset, where the agent inspects the training rows of that dataset.

Both are evaluated with the same protocol: 100 configurations sampled from the space, followed by a greedy ensemble.

Results

Dataset (metric)Base spaceDataset-agnostic, Opus 4.8 (paper)Dataset-agnostic, Opus 5.5Dataset-conditioned, Opus 5.5
churn (accuracy, higher is better)0.85750.85800.85950.8590
MIC (log-loss, lower is better)0.46880.45350.45370.4539
airfoil (RMSE, lower is better)1.04170.96821.00660.9903
california (RMSE, lower is better)0.41570.40790.41550.3833

On three of the four datasets, a dataset-agnostic space is on par with the dataset-conditioned one or better. The exception is California Housing, where the dataset-conditioned space is about 8% better.

What the dataset-conditioned agent writes

Dataset-agnosticDataset-conditioned (4 datasets pooled)
Generated candidates4027 (3 to 10 per dataset)
Feature preprocessing and embeddings13 (33%)16 (59%)
Model architecture8 (20%)2 (7%)
Optimizer and training16 (40%)7 (26%)
  1. The dataset-conditioned agent mostly does feature engineering. With the data in view, it writes features for that dataset and almost stops changing the model. For example, on airfoil all three candidates are feature transforms (dimensionless groups of the physical inputs).
  2. The gain on California comes from spatial features for latitude and longitude. The agent wrote nearest-neighbour target features and a joint two-dimensional embedding of the coordinate pair. California Housing is one of the most studied tabular datasets, and these techniques are published for it. We cannot tell whether the agent derived them from the data or recalled them, which is the contamination issue discussed in our first reply.
  3. Leakage is not only a matter of pre-training. In several runs, we observed the agent retrieving public code and papers with tuned pipelines for the target dataset from the web.
  4. Cost grows with the number of datasets. Each dataset-conditioned generation took about as long as the single dataset-agnostic one (about 40 minutes), so the four datasets cost four times as much, and our 65 datasets would cost 65 times as much.

To summarize

Outside of California, the dataset-agnostic space achieves comparable gains at a fraction of the cost, and it does so without access to the data. The two approaches are also complementary: a dataset-conditioned agent acts as automated feature engineering, and its features can be added on top of our model-level search space. Further, we emphasize that our modularity (splitting pipeline into stages) has helped to guide the agent to produce more diverse candidates. We will add this experiment and its discussion to the paper.

Reviewer_UERzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LLM agents can construct reusable search spaces for tabular machine learning. Given a model family and human-defined module interfaces, an agent proposes, implements, and tests candidate components without access to the target dataset; standard hyperparameter optimization (HPO) then jointly optimizes component choices and hyperparameters. The authors evaluate the approach across five model families and 45 datasets, with additional experiments on TabArena-Lite. Several model families show performance improvements, with more pronounced gains under ensembling, although the benefits vary across models and tasks, and RealMLP does not consistently improve in the single-model setting.

Critical strengths
  • Decoupling component generation from dataset-specific HPO allows the resulting search spaces to be reused across datasets and integrated into existing tuning and ensembling frameworks, making the design practically useful.

  • The experiments span multiple model families and examine candidate-pool size, HPO budget, and repeated search-space generation with different agents, providing substantial evidence on the approach’s applicability and robustness.

Critical weaknesses
  • Given that Text-to-ML already uses LLMs to generate modular search spaces for subsequent AutoML optimization, what substantive novelty does this work offer in constructing reusable within-family search spaces?

  • In the main experiments, the base TabICLv2 model is untuned, whereas its agentic variant undergoes 100 HPO trials; please provide a baseline with a comparable search budget to distinguish the benefits of search-space design from those of additional tuning.

  • Although the human-curated module bank is evaluated using the same HPO and ensembling protocol, the comparison does not control for candidate count, module coverage, or the recency of the included methods; please provide controls matched on these factors and compare construction costs, including human effort, to clarify the agent’s added value in search quality and the efficiency of search-space construction.

  • Table 14 reports a 1.7% single-model improvement and wins on 35/45 datasets for agentic MLP†, whereas the released version in Table 21 achieves 1.28% and a 60% win rate (27/45); please clarify whether these results use different search spaces, experimental runs, or aggregation procedures, and reconcile the reported figures.

Questions for authors

See the weakness.

3 comments

Thank you for your careful review.

Q1. Novelty relative to Text-to-ML.

Thank you for the pointer, we will cite and highlight the differences. Text-to-ML and our method rely on the same concepts: an LLM generates modules, and HPO searches over their combinations. However, Text-to-ML works per-dataset: given a metadata, an LLM generates a complete pipeline for that task, which is then optimized with HPO. Our method generates a search space for a given model, which can be re-used across datasets.

Text-to-MLOurs
Inputmetadatamodel + HP set
LLM outputtask-specific ML pipelinemodules for the given model
LLM generationsone per datasetone per model
LLM seestask description and metadatamodel code
Evaluation12 tasks (tabular, vision, NLP) vs. LLM prompting45 + 51 tabular datasets, vs. SOTA tabular models

Concretely, our method is different from Text-to-ML in the following aspects:

  1. Privacy. Text-to-ML relies on the metadata about the task. In some applications sharing such information is prohibited.
  2. Price. Since Text-to-ML is a per-dataset approach, its cost scales with the number of tasks. We generate modules for each model once offline, and they can be re-used for any new dataset.
  3. Leakage. On public benchmarks, judging Text-to-ML performance is difficult due to potential LLM cheating. Optimal approaches for public benchmark datasets are already known and published, and LLMs can exploit this.
  4. Performance. During the rebuttal, we evaluated TabM-A on the full TabArena (general reply): it has the highest Elo among non-foundation models (1514 [1485, 1547]) and beats TabM on 41/51 datasets (p < 0.001). We note that so far this is the first tabular model on the benchmark, designed with the help of LLM agents.

Q2. TabICLv2 with a matched tuning budget.

We tuned base TabICLv2 over TabArena's search space under the identical protocol (TabArena-Lite, 51 datasets).

EloTabICLv2TabICLv2-AΔ\Delta
Default156115610
Tuned16001616+16
Tuned + ensembled16231632+9

Thank you for suggesting this ablation. With a matched budget, TabICLv2-A still scores higher after tuning and after tuning and ensembling (28 of 51), but the gap is narrower. We will add this comparison to the revision and adjust the TabICLv2 wording in the text accordingly.

Q3. Comparison with human-curated bank.

Candidate count. On the 15 datasets, random agentic candidate sets (drawn from all modules, 10/9 for regression and classification) are mostly on par with the human-curated 19-candidate set (3 modules).

MLP$^\dagger$ search spaceMean Δ\Delta over baseAgentic wins vs. bankWilcoxon p
Human bank (19 candidates)+0.88%--
Agentic, 8 random candidates+0.46%4/150.30
Agentic, 16 random candidates+0.07%3/150.05
Agentic, 32 random candidates+0.74%7/150.85
Agentic, 48 random candidates+1.00%5/150.64
Agentic, full pool (57 / 61)+1.38%--

Effort. One generation takes 25-100 minutes and ≈\approx$30 per model family (Table 3). The closest human-engineered counterpart, RealMLP, is a curated modular space that took a full research project (NeurIPS). As we highlight in the general response, in under two hours the agent produced TabM-A, which reaches the highest Elo among non-foundation models and has a lower error than RealMLP on 30/51 datasets of full TabArena.

Scope of our claim. Matching count, coverage, and recency would compare the judgment of an LLM with that of a human on a fixed menu of modules. That is not our claim. We claim that the agent makes a broad, model-specific search space cheap, fast, and re-runnable as LLMs improve, and that this space improves strong tabular models. We will revise the abstract and Appendix D.2 so that the human-bank comparison is not read as superiority of individual agent-written modules over human-chosen ones.

Q4. Table 14 vs. Table 21.

Thank you for catching this. Table 14 was computed on the 34 datasets on which all three human-bank runs had finished, while Table 21 uses all 45. The submitted caption and win counts wrongly stated 45 datasets. On the 34 datasets, agentic MLP$^\dagger$ wins on 20/34 (59%), consistent with 27/45 (60%) in Table 21. We revise the table in the paper and attach it below.

FamilyHuman bank (AutoPyTorch) Δbase\Delta_{\mathrm{base}} (%)WinsAgentic Δbase\Delta_{\mathrm{base}} (%)WinsAgentic vs. bank: wins
MLP$^\dagger$, single-1.6 [-5.8, +0.7]16/34+1.7 [+0.5, +3.2]20/3422/34
MLP$^\dagger$, ensemble+0.2 [-0.6, +1.1]16/34+1.8 [+0.9, +3.1]29/3425/34
TabM$^\dagger$, single-0.6 [-1.9, +0.5]15/34+0.6 [-0.4, +1.7]21/3421/34
TabM$^\dagger$, ensemble+0.6 [-0.6, +1.8]17/34+1.5 [+0.5, +2.7]24/3421/34
RealMLP, single+0.6 [-0.2, +1.6]17/340.0 [-0.6, +0.5]15/3416/34
RealMLP, ensemble-0.5 [-1.3, +0.0]12/34+0.3 [-0.2, +0.9]20/3421/34

Dear Reviewer UERz,

Thank you again for your review. As the discussion period is coming to an end, we would like to ask whether our reply has addressed your concerns. In brief:

  • Text-to-ML (Q1): we added a comparison and now discuss it in the paper. It runs per dataset, while our agent runs once per model without access to any data.
  • TabICLv2 at a matched budget (Q2): we ran the ablation that you suggested and added it to the paper.
  • Human-curated bank (Q3): we clarified the scope of our claim and revised the abstract and Appendix D.2 accordingly.
  • Table 14 vs. Table 21 (Q4): thank you for catching this. The corrected table is in our reply and in the paper.

We also evaluated TabM-A on the full TabArena, where it has the highest Elo among non-foundation models. If any point remains open, we would be glad to answer it in the remaining time.

Dear Reviewer UERz,

We are writing once more as the discussion period closes soon. Since our reply, we have added two results that relate to your questions (details in the general follow-up):

  • A new model. We applied the method, without changes, to TabPack. On TabArena-Lite, TabPack-A gains +70 Elo over TabPack (95% CI [+8, +125]) and has a lower error on 36 of the 51 datasets.
  • Dataset-conditioned generation. On three of four datasets, our dataset-agnostic space is on par with spaces generated per dataset, at a fraction of the cost. This supports the difference from Text-to-ML that we describe in Q1.

Our reply also covers your four points: the comparison with Text-to-ML, TabICLv2 at a matched budget, the scope of the human-bank comparison, and the corrected Table 14.

We would be grateful if you could let us know whether these address your concerns, or which point remains open.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Reply

We thank all reviewers for their time and comments. Below, we highlight our contribution and share new results. Specifically, TabM-A, i.e. TabM with the agentically-expanded HP set, has achieved state-of-the-art performance on the TabArena benchmark among non-foundation models. We answer each reviewer's questions in the individual replies.

Problem

We study how to use LLMs for HPO in tabular ML efficiently and economically. Tabular data is used everywhere, yet the field has so far gained little from LLM agents: to our knowledge, none of the established models and AutoML systems on TabArena or other benchmarks uses an LLM in its tuning pipeline, except for helping to handle strings.

Difficulties

  • Tabular ML is cheap, and frontier LLMs are not. A tuned LightGBM runs on a laptop in minutes. Many times, using expensive LLMs for tabular tasks is considered not economical.
  • The best configuration depends on the dataset. Tabular datasets differ widely in size, feature types, and noise. We find that using LLMs by naively asking it to improve the model in an autoresearch loop does not work due to the interference from bad hyperparameters.
  • Potential dataset leakage and generalizability. It is hard to separate LLM memorization from skill. LLMs can recognize datasets from column values alone. Public benchmark datasets have been utilized extensively in various studies and optimal techniques for each are available online.
  • Privacy. Further, there is another problem of exposing data to LLMs, because in many applications, this can be strictly prohibited.

Our method

The agent and HPO each specialize in what they do best.

  • The agent writes code once per model. It proposes modifications from the literature, implements them as modules (preprocessing, embeddings, etc), and tests them on synthetic data. It never sees the target dataset or metadata, so data leakage is excluded and privacy guaranteed.
  • HPO selects per dataset. A standard HPO algorithm chooses the combination of modules and standard hyperparameters that works best on each dataset.
  • No LLM at deployment. We release the search space with all generated modules. On a new dataset, users run standard HPO.
MethodAgent runsAgent API costAgent sees the dataLLM needed on a new dataset
AIDE, SELA, AutoML-Agentper dataset-yesyes
Autoresearch (our baseline, Table 1)per dataset$20-50 per datasetyesyes
TabFM-Auto (Fu et al., 2026, concurrent)once per dataset$17.6K in total (≈\approx$70 per dataset)yes, with metadatayes
Oursonce per model family (1 generation)≈\approx$30 per generationnono

New results.

TabM-A on full TabArena

Full TabArena (51 datasets, 816 splits), tuned + ensembled. In time for this review, we evaluated TabM-A under the full protocol. It reaches the highest Elo among non-foundation models, ahead of RealMLP, AutoGluon best_quality, and the official TabM. Non-foundation models remain important in practice because they are fast at inference, require less computational resources, and scale to large datasets.

ModelElo95% CI
TabM-A (ours)1514[1485, 1547]
AutoGluon best_quality1484[1443, 1525]
RealMLP1478[1437, 1517]
TabM1425[1390, 1469]
LightGBM1408[1381, 1433]
CatBoost1399[1365, 1431]

TabM-A has a lower error than RealMLP on 30/51 datasets, than AutoGluon on 32/51, and than the official TabM on 41/51 (Wilcoxon p < 0.001).

Early signs of LLM progress

We have also run module generation with the newest Claude Opus 5.5 (~$30 and 45 minutes for generating modules). We report its results on TabArena-lite (first outer train-test split). Although TabArena-lite is noisy, we see good potential for the model, as it outperforms both our old version of TabM-A (generated with Opus 4.8, which claims SoTA on the full TabArena) and base RealMLP.

ModelDefaultTunedTuned + ensembled
TabM-A-O55125714391514
RealMLP123914191482
TabM-A126014041474
TabM126913621396
LightGBM114813331385
CatBoost133513891382

We highlight the differences between the generated modules by the two LLMs below.

ModuleTabM-A (Opus 4.8)TabM-A-O55 (Opus 5.5)
Generated candidates5940
OptimizerAdamW, Lion+ Muon
Numerical featuresper-feature transforms+ pairwise arithmetic interactions

Changes in the revision.

  1. The full-TabArena results and the matched-budget TabICLv2 comparison above.
  2. Paired bootstrap CIs and Wilcoxon tests.
  3. A corrected Table 14 on 34 datasets.
  4. A discussion of Text-to-ML, SELA, AIDE, AutoML-Agent, and TabFM-Auto.
  5. Fixed naming throughout the paper and a readable Figure 5.
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank the reviewers for the discussion. Below, we summarize the two experiments that we added during the discussion period.

1. A new model: agentic TabPack

To test that the method transfers to a model outside our submission, we applied it, without changes, to TabPack, and evaluated the result on TabArena-Lite (51 datasets, first outer split).

MethodElo [95% CI]
TabPack-A (ours)1409 [1338, 1491]
TabPack1340 [1263, 1423]

In the paired comparison, TabPack-A gains +70 Elo (95% CI [+8, +125]) and has a lower error on 36 of the 51 datasets. Together with the submission and the general reply, the method now improves six model families (MLP, TabM, RealMLP, LightGBM, TabICLv2, TabPack) and works with three LLMs (Opus 4.8, GPT 5.5, Opus 5.5).

2. Dataset-conditioned generation (requested by Reviewer BhxW)

For TabM$^\dagger$, we compared our dataset-agnostic space with spaces that the agent generates per dataset after inspecting the training rows. Details are in our reply to Reviewer BhxW.

Dataset (metric)Dataset-agnostic, Opus 4.8 (paper)Dataset-agnostic, Opus 5.5Dataset-conditioned, Opus 5.5
churn (accuracy, higher is better)0.85800.85950.8590
MIC (log-loss, lower is better)0.45350.45370.4539
airfoil (RMSE, lower is better)0.96821.00660.9903
california (RMSE, lower is better)0.40790.41550.3833
  • On three of the four datasets, a dataset-agnostic space is on par or better. The exception is California Housing.
  • With the data in view, the agent mostly does feature engineering. 59% of its candidates are feature transforms or embeddings (33% for the dataset-agnostic generation), and only 7% change the model (20% resp. in dataset-agnostic).
  • Contamination cannot be ruled out. The gain on California comes from spatial features for latitude and longitude, which are published for this dataset.
  • The cost grows with the number of datasets. Each per-dataset generation takes about as long as our single generation for all datasets.

Summary of the discussion period

  1. TabM-A on the full TabArena: the highest Elo among non-foundation models (general reply).
  2. TabM-A generated with Opus 5.5: a further improvement on TabArena-Lite (general reply).
  3. TabPack-A: the method transfers to a new model (above).
  4. Dataset-conditioned generation: comparable results on three of four datasets, at a multiple of the cost (above).
  5. Paired significance tests, the matched-budget TabICLv2 comparison, and the corrected Table 14 (individual replies).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.