- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLM agents can improve tabular machine learning by expanding the hyperparameter optimization (HPO) search spaces of existing models. The authors decompose each model into modular components, such as preprocessing, embeddings, architecture, training, and inference. An LLM agent proposes and implements alternative methods for these components, which are then jointly optimized with the original hyperparameters using classical HPO.
The approach is evaluated on five tabular model families across 45 datasets, with additional experiments on TabArena. The results show improvements over the original search spaces, particularly on small-to-medium regression tasks and in ensemble settings. The paper also examines agent generation stability, candidate pool size, and comparisons with existing AutoML approaches.
Critical strengths
-
The paper addresses a practical problem in tabular machine learning. Separating agent-based candidate generation from classical HPO is a sensible design, as it allows the generated search space to be reused across datasets without repeatedly invoking the agent.
-
The experimental evaluation is relatively comprehensive, covering multiple model families, datasets, and evaluation protocols. The authors also provide useful analyses of candidate diversity, search-space size, and stability across different agents and independent generations. These experiments strengthen the empirical findings.
Critical weaknesses
-
Insufficient discussion and comparison with prior agent-based AutoML methods. The paper does not adequately position its contribution relative to existing LLM-agent-based AutoML approaches, such as SELA [1], AutoML-Agent [2], and AIDE [3]. In particular, SELA already explores using LLM-generated suggestions to construct search spaces across different ML pipeline stages, followed by search-based optimization. While the proposed method differs by generating reusable, model-specific modules and delegating optimization to classical HPO, these distinctions deserve a more thorough discussion. Moreover, the experimental comparisons focus primarily on traditional AutoML systems, while the only agent-based baseline is an end-to-end autoresearch loop evaluated on four datasets. Although directly comparing these frameworks may require adapting their settings, a more comprehensive discussion and comparison with existing agent-based approaches would help clarify the contributions and practical advantages of the proposed method.
-
Limited isolation of the sources of performance improvements. The comparison with the human-curated AutoPyTorch component bank is informative, but the two search spaces differ substantially in size and coverage. For example, the human-curated bank lacks numerical embedding and target preprocessing alternatives included in the agent-generated space. Since Section 5.3 also shows that performance improves as the candidate pool grows, it remains unclear how much of the improvement comes from the effectiveness of agent-generated candidates rather than the broader coverage of existing techniques. A more closely matched comparison would help clarify this. Additionally, the TabICLv2 baseline is evaluated without HPO, whereas its agentic variant receives 100 tuning trials. This makes it difficult to distinguish the benefits of search-space expansion from those of additional tuning.
[1] Chi, Y., et al. (2024). SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning. arXiv:2410.17238.
[2] Trirat, P., Jeong, W., & Hwang, S. J. (2025). AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. ICML 2025. Paper.
[3] Jiang, Z., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.
Questions for authors
-
Could the authors further clarify how the proposed framework differs from existing agent-based AutoML approaches? In particular, what do the authors consider the main advantage of generating reusable search spaces over existing agent-driven pipeline optimization methods?
-
Section 5.3 shows that performance generally improves as the candidate pool grows. To what extent do the authors believe the improvements come from the quality of agent-generated candidates, rather than simply having a larger and more diverse search space? Are there any observations from the experiments that could help distinguish these factors?
-
For TabICLv2, the original model is evaluated without HPO, while the agentic variant receives additional tuning trials. Could the authors clarify how much of the observed improvement is attributable to the proposed search-space extensions, as opposed to the benefit of tuning itself?
5 comments
Thank you for the response and for providing the additional comparison with TabICLv2.
However, I still have questions regarding the comparison with existing AutoML component libraries. I notice that most of the components proposed by the agent are general-purpose; how does this differ from simply writing additional plug-ins for existing AutoML systems? Furthermore, some of the baselines do not use the latest versions—for instance, AutoKeras 1.1.0 is used instead of 3.0.0. While there may be a valid reason for this choice, it is not explained in the paper.
I would also like to gain a deeper understanding of why this approach is effective. While the cross-dataset results demonstrate the utility of an expanded search space, it is perhaps not surprising that a set of general-purpose components can be applied across different datasets. Are there any specific findings that highlight what makes the components or combinations proposed by the agent unique? Otherwise, it seems plausible that similar gains could be achieved by simply incorporating such components into other AutoML systems. Given the differences in scope, the current comparison with manually curated component libraries does not yet fully address this point.
Consequently, I remain reserved regarding the paper’s contributions and will maintain my current score.
Thank you for the follow-up. Your argument, as we understand it, is: (1) the agent's components are general-purpose, hence (2) they could be added to any AutoML system as plug-ins, hence (3) the gain is not specific to our method. We address each step.
1. Many components are not general-purpose
The agent writes them for a given model, after reading its code.
| Model | Examples of generated candidates |
|---|---|
| Any neural model | Muon, Lion, SAM, and Lookahead optimizers, SWA, self-distillation |
| TabM | low-rank (LoRA-style) per-member adapters on the shared weights, a diversity loss between the ensemble members, variance-weighted aggregation of the member predictions |
| RealMLP | BatchEnsemble layers in place of its independent ensemble members, shared categorical embedding tables with per-member adapters, residual blocks with RealMLP's data-dependent initialization |
| TabICLv2 | feature-permutation schemes and aggregation rules for its in-context ensemble |
2. Hence they cannot simply be plugged into any AutoML system
A per-member adapter needs TabM's shared-weight ensemble, and a data-dependent initialization needs RealMLP's training procedure. AutoKeras and AutoPyTorch search over their own backbones, so these candidates have nothing to attach to. The unit that transfers is the model together with its space. Where a system contains the model, the transfer works, and we show it: AutoGluon (+A) receives our agentic RealMLP and TabM$^\dagger$ spaces and improves on AutoGluon on all four aggregate metrics (Figure 2).
3. Why the approach is effective
The method works because three facts hold together.
- Tabular tasks are heterogeneous. TabArena contains biological, financial, real-estate, and engineering datasets, and each kind favors different components. For example, engineering targets often depend on ratios and products of the input features, and real-estate prices depend on location, for which piecewise-linear embeddings are effective.
- Hence the gain comes from per-dataset selection, not from one component. No candidate wins on more than 12 of the 45 datasets, and every candidate wins on at least one (Figure 3).
- The agent supplies many diverse candidates. In our experiments, a larger pool of candidates generally helps. The agent also follows recent literature: the Opus 5.5 generation added the Muon optimizer and pairwise arithmetic feature interactions, both from 2025-2026 papers, and it improves on TabM-A on TabArena-Lite (Elo 1514 vs. 1474).
- Standard HPO finds the right combination for each dataset. Unlike per-dataset agent loops, the search over the agent's candidates needs no LLM.
As a result, a single model outperforms a full AutoML system. On the full TabArena, TabM-A has a lower error than AutoGluon best_quality on 32/51 datasets and the highest Elo among non-foundation models.
AutoKeras version
Thank you for noticing this. We ran AutoKeras 1.1.0. Version 2.0.0 removed the tabular tasks, and we overlooked that 3.0.0 restored them. We have now compared the two releases. The tabular search space is the same in both: the architecture, the optimizer choices, and the tuner with its initial configurations. The implementation differs in three places:
| 1.1.0 | 3.0.0 | |
|---|---|---|
| Categorical features | IDs assigned by frequency | IDs assigned in sorted order |
| Training | AdamW with warmup | AdamW without warmup |
| Model used for prediction | final refitted model | best checkpoint of the search |
We will rerun the AutoKeras baseline with 3.0.0 and report it in the paper.
Question
AutoGluon (+A) is the comparison that you describe for a system that contains our models. Could you tell us which additional comparison would address the remaining difference in scope?
Response
Thank you for the detailed review and the clear questions.
Q1: Difference from agent-based AutoML (SELA, AutoML-Agent, AIDE).
We will add a discussion of these works. They solve a broader task: given a dataset and a prompt, build a full pipeline for that dataset. Our task is narrower: given a model implementation, agent writes a search space that improves it, once and offline. On a new dataset, the user runs standard HPO over the released modules.
Q2. Main advantage of our method.
Our method is particularly attractive for engineers and researchers working on tabular data, because:
best_qualityQ3: Quality and quantity of the agent-generated candidates and comparison with human-curated banks
We see the size and coverage of the space as the benefits that an LLM agent can provide over traditional (human-engineered) AutoML systems. The practioners are untied from package (AutoKeras, AutoPyTorch) releases. For example, a re-generation of the TabM modules with the newer Opus 5.5 added the Muon optimizer and arithmetic feature interactions, and improves on TabM-A on TabArena-Lite (Elo 1514 vs. 1474, general reply).
Two observations speak to the quality of the candidates. First, no single candidate carries the gain: within the most important modules, no candidate wins on more than 12 of the 45 datasets, and every candidate wins on at least one (Figure 3). Second, the quality is reproducible: ten independent re-generations by two agents (Claude Code and Codex) all improve over the base MLP$^\dagger$ space on all four aggregate metrics (Table 4).
Q4: TabICLv2 tuning.
We tuned base TabICLv2 over TabArena's own search space (TabArena-Lite, 51 datasets).
Thank you for suggesting this ablation. With a matched budget, TabICLv2-A still scores higher after tuning and after tuning and ensembling (28 of 51), but the gap is narrower. We will add this comparison to the revision and adjust the TabICLv2 wording in the text accordingly.
References
[1] Fu et al. TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models. arXiv:2609.37989, 2026.
[2] Gorishniy et al. Revisiting Deep Learning Models for Tabular Data. NeurIPS 2021.