acceptodds
Under review as a conference paper at ICLR 2027

LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search

Abstract

Neural Architecture Search (NAS) aims to automatically discover high-performing deep neural network (DNN) architectures. However, conventional algorithm-driven NAS relies on carefully hand-crafted search spaces to ensure executability, which restricts open-ended exploration. Recent coding-based agentic approaches using large language models (LLMs) reduce manual design, but current LLMs struggle to reliably generate complex, valid architectures, and their proposals are often biased toward a narrow set of patterns observed in their training data. To bridge reliable algorithmic search with powerful LLM assistance, we propose LLMasTool, a hierarchical tree-based NAS framework for stable and open-ended model evolution. Our method automatically extracts reusable modules from arbitrary source code and represents full architectures as hierarchical trees, enabling evolution through reliable tree transformations rather than code generation. At each evolution step, coarse-level planning is governed by a diversity-guided algorithm that leverages Bayesian modeling to improve exploration efficiency, while the LLM resolves the remaining degrees of freedom to ensure a meaningful evolutionary trajectory and an executable generated architecture. With this formulation, instead of fully agentic LLM approaches, our method explores diverse directions beyond the inherent biases in the LLM. Our method improves over existing NAS methods by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating its effectiveness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Architecture Search, NAS, Agentic NAS, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FMMtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes the LLMasTool neural architecture search framework, which no longer allows large models to directly generate complete networks, but instead uses LLM as a local decision-making tool. The method combines source code module mining, hierarchical architecture tree, and coarse to fine evolution strategy, relying on predefined tree transformation operations to modify the structure, and using hierarchical Thompson sampling selection operations and prompt categories. Compared to NADER, we achieved stable accuracy improvement on the CIFAR and ImageNet datasets under a 500 search budget; The paper is accompanied by various components such as rich ablation, validation tree transformation, module library, sampling hierarchy, model size, and training rounds.

Critical strengths

The paper conducted a complete and controllable component ablation experiment with solid experimental evidence. The results show that tree structure editing is superior to direct code generation, and the mined reusable module library has significant effects. Hierarchical Thompson sampling has a positive effect on search; At the same time, stability was verified under different LLM parameter quantities and longer training cycles. The entire system does bring observable performance improvements, and its practical value cannot be denied just because it is built on existing technology.

Critical weaknesses

There are two core weaknesses. Firstly, the paper focuses on the discovery of an open architecture that breaks through the manual search space. However, the actual search space is still limited by fixed module libraries and preset tree operations, and there is no evidence to suggest that it can generate new computing primitives outside the module library or rewrite the internal computing logic of the module; The experiment can only prove that module combination reuse has good effects, and it is difficult to support the proposition of "open discovery". Secondly, there is a lack of controlled strong agent comparison: only different sizes of Qwen3 were tested within the same protocol, without comparison with other large model agents such as GPT and DeepSeek under the same budget and module conditions. Therefore, the current advantage cannot be extended to other LLM systems.

Questions for authors

The main motivation for the paper is to break through the manual search space limitations of traditional NAS, but LLMasTool still relies on pre built module databases and fixed tree transformation operations. Although this approach can increase the flexibility of combining existing modules, its essential difference from traditional module level NAS is not yet clear.

Please further explain: What new architectural structures can this method explore beyond existing module combinations and parameter adjustments? Is there any specific experiment that can prove its ability to discover new computing structures not included in the module library? If the search mainly relies on the recombination of existing modules, how does the author define the specific meaning and applicable boundaries of * open-ended NAS * in the paper?

The paper tested Qwen3 models of different scales in the appendix, but has not yet compared other model families and stronger agent-based architecture search methods under unified experimental conditions. Considering that the core viewpoint of the paper is to use LLM as a local decision-making tool, rather than allowing it to autonomously generate a complete network, this comparison is more important for verifying the advantages of the method.

Can the author explain whether the complete Agent method using GPT, DeepSeek, and other models is still significantly weaker than LLMasTool under the same search budget, module resources, execution checks, and feedback mechanisms? Can the existing results distinguish the benefits brought by structured search strategies from the impact caused by differences in underlying LLM capabilities?

Table 3 shows that using mining module libraries can bring significant performance improvements compared to basic module libraries, while Repeat Previous and Hierarchical Thompson Sampling in Table 2 also have a significant impact. Therefore, the current results cannot fully distinguish the independent contributions of module resources, architecture representation, and search strategies.

Can the author further explain how much independent benefit does Hierarchical Thompson Sampling have compared to other conventional search strategies, given fixed module databases, tree transformation operations, and search budgets? In addition, can the main results of 500 searches in Table 1 provide the mean and standard deviation of multiple independent runs to verify the stability of performance improvement?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FMMt Part 1

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the reviewer’s recognition of the effectiveness of our component ablations, the benefits of the mined module library, and the positive contributions of hierarchical Thompson sampling. We agree that the scope of open-ended NAS, the generality across different LLM backbones, and the independent contributions of individual components deserve further clarification. We address these concerns below.

Clarification of Open-Ended NAS and the Difference from Conventional Module-Level NAS

We agree that the distinction between LLMasTool and conventional module-level NAS needs further clarification. In LLMasTool, open-endedness refers to the ability to expand the architectural search space beyond manually predefined architecture templates through automatically mined modules and hierarchical structural compositions. LLMasTool can automatically construct a module database from arbitrary PyTorch source code. Furthermore, by leveraging the LLM's ability to understand source code, it can automatically identify valid module combinations, as well as tunable hyperparameters and their feasible ranges. As a result, the combinations of modules and hyperparameter configurations give rise to an almost infinite, open-ended search space.

Conventional NAS search spaces tend to become outdated over time as new architectural designs emerge. LLMasTool addresses this limitation by allowing newly proposed modules from other research papers to be incorporated simply by adding their source code to the module database. This enables LLMasTool to continuously propose architectures that incorporate the latest advances in the field without requiring manual redesign of the search space. We believe this is one of the major advantages of LLMasTool over existing NAS methods.

Although the current framework does not support the proposal of low-level computational primitives, such as custom CUDA kernels, its "create module operation" enables the creation of new modules, similar to those designed by researchers and engineers. In fact, LLMasTool has already discovered several interesting module designs that, to the best of our knowledge, have not been reported in previous studies.

For example, Lines 2782–2827 of the architecture discovered for CIFAR-100 reveal a module immediately following the stem layer in which an Attention branch and a CNN branch are processed in parallel, and their outputs are merged through element-wise multiplication, as shown below.

             Input
               |
      +--------+--------+
      |                 |
    Conv2d            Conv2d
      |                 |
     GELU              GELU
      |                 |
     AdditiveAtten   Linear
      |                 |
      +--------+--------+
               |
Element-wise Multiplication
               |
             Output  

Interestingly, similar structures are also found in the architectures discovered for CIFAR-10 and ImageNet16-120. This suggests that such parallel Attention-CNN modules may be particularly effective when placed immediately after the stem layer. We believe this observation could provide useful insights for future architecture design. Furthermore, the best architecture discovered on ImageNet16-120 incorporates an even more complex module, demonstrating LLMasTool's ability to discover better module structures.

AuthorsOpenReview ↗

Response to Reviewer FMMt Part 2

Comparison Across LLM Families and Fully Autonomous Agent

We appreciate the reviewer’s concern that evaluating different Qwen3 model sizes alone is insufficient to establish the merits of LLMasTool against powerful LLM and agentic frameworks.

To investigate this, we conducted additional experiments using different LLM, including closed LLM of GPT6.1-Terra and newer open LLM of Qwen3.8-27B. We used Github Copilot CLI version 1.0.112 for a agentic harness. The results are summarized below. Validity rate represents the proportion of generated architectures that pass the feasibility check and successfully run. #Tokens denotes the average number of tokens (input + output) required to generate one valid, executable architecture, including tokens consumed by unsuccessful attempts.

LLMevo. stepFramework/Harness#TokensValidity RateAccuracy
Qwen3-8Bcode-gen-71,00012.828.62
Qwen3-8Btree-transLLMasTool48,00023.440.78
Qwen3.8-27Bcode-gen-45,00020.130.59
Qwen3.8-27Bcode-genCopilot2,600,00032.234.34
Qwen3.8-27Btree-transLLMasTool18,00053.339.89
GPT6.1-Terracode-genCopilot682,00078.132.42

Interestingly, NAS performance tends to deteriorate when using LLMs that are generally considered more capable. Investigating the underlying reasons is left for future work. As discussed in Appendix C.5, one possible hypothesis is that more capable LLMs may exhibit stronger prior biases, potentially limiting their ability to explore diverse trials. For this reason, simply using a more capable LLM or an agentic harness cannot serve as a substitute for LLMasTool.

Furthermore, when using the same underlying LLM, LLMasTool substantially outperforms both direct LLM-based NAS and agentic harnesses such as Copilot, demonstrating the effectiveness of our proposed framework.

AuthorsOpenReview ↗

Response to Reviewer FMMt Part 3

Independent Contributions of Architecture Representation, Module Database, and Search Strategy

We conducted an ablation study in which the main proposed components were incrementally added on ImageNet-100 dataset.

representationmoduleplanningrepeatsamplingaccuracy
code----28.62
treeprimitiveLLM's ideaw/ouniform30.24
treeminedLLM's ideaw/ouniform33.45
treeminedc-to-fw/ouniform36.18
treeminedc-to-fw/uniform38.86
treeminedc-to-fw/H-Thompson40.78

As the results show, each of the proposed components contributes positively to the overall performance.

Unfortunately, we did not have sufficient time to repeat all ablation experiments multiple times and report the standard errors. However, we plan to conduct these additional experiments before the camera-ready submission to provide more reliable results. For some ablation studies investigating the effectiveness of the sampling strategies, we have already completed three independent runs. Please refer to our response to Reviewer rjWE (Stability of Planning Strategies) for these results.

Statistical Significance and Stability Across Independent Runs

We appreciate the reviewer’s suggestion to report the mean and standard deviation of the main results of 500 searches. The results of three runs are reported below.

MethodCIFAR-10 valCIFAR-10 testCIFAR-100 valCIFAR-100 testImageNet16-120 valImageNet16-120 test
NADER91.5594.6275.7276.0050.2050.52
LLMasTool93.00 ± 0.1895.20 ± 0.1877.57 ± 0.2577.82 ± 0.1552.39 ± 0.2053.18 ± 0.48

With 500 search trials, LLMasTool achieves even more stable performance than with 100 trials. These results show that LLMasTool consistently outperforms previous methods.

Reviewer_ixWVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper combines code mining, tree transformations, and hierarchical sampling. It uses these techniques for LLM-assisted neural architecture search. The method is evaluated on four image classification datasets. However, the empirical evaluation has substantial gaps. The claimed contributions are also not clearly distinguished from what the implementation actually supports. These issues warrant further revision.

Critical strengths

The proposed module mining and hierarchical tree offer practical engineering value. On all three datasets in NAS-Bench-201 (NB201), the proposed algorithm reports higher test accuracy than NADER. This shows the potential of open-ended architecture search.

Critical weaknesses

1.The paper has too few substantive experiments. The appendix is not concise. Key points are hard to find. 2.Training-free NAS is underdiscussed. Add representative methods. Clarify the search-cost advantages and limits. 3.The results do not yet prove SOTA for LLMasTool. Only the NB201 space is further optimized. DARTS and Transformer spaces are untested. Cross-space applicability remains unclear. 4.There is no ImageNet-1K validation. Small-scale classification is not enough to support the conclusions. 5.LLM search on NB201 is promising. But results are below the 2019 ICLR DARTS paper on CIFAR-10 and CIFAR-100. ImageNet-1K is also missing. 6.The authors should show gains on three standard NAS search spaces.

Questions for authors

How does LLMasTool compare with training-free NAS in accuracy and cost? What are its advantages and limitations? Please add experiments on DARTS cell or Transformer search spaces, and on ImageNet-1K. This would help verify cross-space applicability and scalability.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ixWV Part 1

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate the recognition of the practical value of our module mining and hierarchical tree representation, as well as the promising results on NAS-Bench-201.

We agree that further evidence on generality and scalability, as well as a clearer discussion of training-free NAS, would strengthen the paper. We address these points below with additional experiments and clarifications, organized around four aspects: (1) comparison with training-free NAS, (2) applicability to different NAS search spaces, and (3) scalability to ImageNet-1K. All additional results and discussions, together with the corresponding references, will be incorporated into the revised manuscript.

Discussion on training-free (zero-shot) NAS Methods

We agree that Zero-shot NAS is an important category of NAS methods and should be discussed more thoroughly. First, our proposed method is a framework for generating architectures and is therefore orthogonal to existing zero-shot NAS metrics, which evaluate the performance of generated architectures. To investigate whether existing zero-shot NAS metrics can be effectively combined with our framework, which operates in an open-ended setting, we evaluated their ability to correctly rank the architectures generated during the CIFAR-100 search using Kendall's rank correlation coefficient.

[Ranking Correlation of Zero-Shot NAS Metrics on Architectures Generated by LLMasTool]

MethodKendall's tau
GraSP [1]0.09
Fisher [2]0.19
SNIP [3]0.20
SynFlow [4]0.22
NEAR [5]0.31
ZiCo [6]0.38
MeCo$_{opt}$ [7]0.60
FLOPs0.57

As shown in the table above, most existing zero-shot metrics rank architectures worse than simple FLOPs count, and even the best-performing one (MeCo) is only marginally better.. Therefore, they are currently difficult to use as reliable evaluation functions for architecture selection. We believe this limitation stems from the challenges posed by open-ended search spaces, such as that explored by LLMasTool. Existing zero-shot NAS metrics are typically evaluated on relatively narrow search spaces, where architectures are constructed by combining a fixed set of primitive modules, resulting in limited architectural diversity. In contrast, LLMasTool explores architectures composed of diverse modules. This highlights the need for more general-purpose zero-shot NAS metrics that can consistently evaluate architectures across diverse module types in open-ended search spaces.

[1] Picking winning tickets before training by preserving gradient flow, ICLR 2020
[2] “Blockswap: Fisher-guided block substitution for network compression on a budget, ICLR 2020
[3] SNIP: Single-shot network pruning based on connection sensitivity, ICLR 2019
[4] Pruning neural networks without any data by iteratively conserving synaptic flow, NeurIPS 2020
[5] NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance, ICLR 2025
[6] ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients, ICLR 2023
[7] MeCo: Zero-Shot NAS with One Data and Single Forward Pass via Minimum Eigenvalue of Correlation, NeurIPS 2023

AuthorsOpenReview ↗

Response to Reviewer ixWV Part 2

Comparison with Different NAS Search Spaces

We agree that evaluating the method under diverse settings is essential to demonstrate its generality. However, we would like to emphasize that LLMasTool is not designed to operate within a fixed, predefined search space, but rather to discover high-performing architectures from an open-ended search space. Although Table 1 provides a fair comparison with other open-ended NAS methods, comparisons with methods operating in the hand-crafted NAS-Bench-201 search space are limited to matching the target models in terms of parameter count and FLOPs, as the underlying search spaces are fundamentally different.

Nevertheless, to further evaluate LLMasTool under a conventional NAS setting, we conducted an additional experiment in which we generated 100 architectures while matching the DARTS training protocol (600 epochs) and imposing a parameter budget of at most 4.5M.

[Performance Comparison under a Matched Training Protocol and Parameter Budget of DARTS]

MethodCIFAR-10 test errorCIFAR-100 test error# MParams
NASNet [1]2.6317.813.3
IS-DARTS [2]2.56-4.3
NCD-NWOT [3]2.4416.484.5
LLMasTool2.3815.984.3

Our method consistently outperforms existing approaches that search within manually designed, hand-crafted search spaces, under the same DARTS training settings and comparable model sizes.

We acknowledge that evaluating target models is computationally expensive in open-ended search spaces, as one-shot NAS and zero-shot NAS are not directly applicable. However, we believe the key advantage of our approach lies in eliminating the need for manual search space design and enabling the flexible incorporation of state-of-the-art modules. This allows LLMasTool to discover better architectures by leveraging the latest advances in architectural components, rather than being constrained by predefined search spaces. We consider this flexibility and adaptability to evolving architectural innovations a major advantage of open-ended NAS.

[1] Learning Transferable Architectures for Scalable Image Recognition, CVPR 2018
[2] IS-DARTS: Stabilizing darts through precise measurement on candidate importance, AAAI 2024
[3] Beyond the Limits: Overcoming Negative Correlation of Activation-Based Training-Free NAS, ICCV 2025

Evaluation on ImageNet-1K

We agree that ImageNet-1K is an important benchmark for evaluating the practical effectiveness of NAS methods at scale and to compare with human researchers.

To address this concern, we extended our evaluation to ImageNet-1K. Evolution was initialized from the EdgeNeXt-XXS model, and 100 architectures were generated and evaluated using the evaluation metric at epoch 100 as the fitness score.

[Performance Comparison on ImageNet-1K]

MethodMFLOPs# MParamsTop-1 Accuracy
MobileNetV3-Small [4]602.567.7
MobileOne-S0 [5]2752.171.2
EdgeNeXt-XXS [6]2601.371.2
FasterNet-T0 [7]3403.971.9
LLMasTool3001.972.0

Given the limited time available, we present the ImageNet-1K experiment as a preliminary result to demonstrate that LLMasTool can be successfully applied to a large-scale benchmark and discover architectures that achieve competitive performance compared with manually designed efficient models. We acknowledge that the current comparison is not aligned with other NAS methods in terms of training and evaluation settings. We will conduct additional experiments under strictly matched settings and include a more comprehensive comparison in the revised version.

[4] Searching for MobileNetV3, ICCV 2019
[5] MobileOne: An Improved One millisecond Mobile Backbone, CVPR 2023
[6] EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications, ECCVW 2022
[7] Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks, CVPR 2023

Reviewer_rjWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LLMasTool, a neural architecture search framework that combines algorithmic exploration with localized LLM assistance. It mines reusable modules from PyTorch repositories, represents architectures as hierarchical trees, and evolves them through structured edits. Hierarchical Thompson sampling selects coarse transformation directions, while an LLM resolves the remaining choices. Experiments cover CIFAR-10, CIFAR-100, ImageNet16-120, and ImageNet-100 under parameter and FLOP constraints. With 500 architecture trials, the method reports test accuracies of 95.31%, 77.83%, and 53.20% on the first three datasets. Ablations examine module mining, architecture representation, planning strategies, and the extent of LLM involvement.

Critical strengths

I like the idea of letting the search algorithm choose where to explore while using the LLM to resolve implementation details. The tree representation makes edit locations and module choices explicit, and mining existing implementations provides a practical way to expand the available building blocks. This gives the framework more control over architectural changes while still benefiting from the LLM’s ability to interpret unfamiliar module code.

Critical weaknesses

Matching parameter and FLOP limits does not match the available building blocks. Table 3 shows that the mined library alone substantially changes performance, while many Table 1 baselines operate in much narrower spaces. For the code/config-generation ablations, it is unclear whether the baselines receive the same mined modules and source-code context. These controls matter for attributing gains to the proposed division of decisions. Moreover, the 500-trial headline results and key ablations lack reported multi-run uncertainty, making the strength of that attribution harder to judge.

The paper emphasizes reliable tree transformations, but several operations still ask the LLM to generate a complete configuration, including hyperparameter changes, removal, and Repeat Previous (Appendix D). Algorithm 1 then filters candidates through feasibility checks; filtering invalid outputs does not establish that generation itself is reliable. I could not find validity rates, rejection counts, or a clear statement of whether rejected proposals consume the search budget. Without these measurements, it is difficult to assess the claimed reliability advantage or compare the actual cost of obtaining 100 valid architectures.

Questions for authors
  1. Do the code/config-generation baselines receive the same module library, source-code context, LLM, and feasibility checks? Could you clarify these controls and distinguish gains from better building blocks versus better search decisions?

  2. How stable are the key planning ablations across independent searches? In particular, does hierarchical Thompson sampling consistently improve over uniform sampling and LLM-led planning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rjWE Part 1

We sincerely thank the reviewer for the positive assessment of our central idea, particularly the separation of algorithmic exploration from localized LLM assistance, as well as the explicit tree representation and reusable module mining. We appreciate insightful comments on experimental fairness, statistical robustness, and the reliability of LLM-generated configurations. We address these concerns point by point below.

Fairness of Comparisons and Contribution of the Proposed Search Strategy

We agree that controlling the search space and experimental conditions is essential for isolating the contribution of our proposed search strategy.

First, regarding the code/config-generation baselines in Table 2, it was impractical to provide the entire source code of the module database to the LLM at once due to context-length limitations. Therefore, the baselines did not have access to the module database and instead relied solely on the LLM's internal knowledge to perform architecture evolution.

To further assess the effectiveness of LLMasTool against code-generation with the access of module database, we additionally compare three types of code-generation approaches against our proposed method.

Knowledge-based Generation: As in Table 2, the LLM generates source code based solely on its internal knowledge, without access to external knowledge such as the module database.

Module Generation: To improve upon direct code generation, we introduce module-level code generation. Following LLMasTool, we randomly identify a module to improve and then mutates that module by generating new source code. In 50% of the cases, the LLM is simply prompted to devise an improvement independently. In the remaining 50%, we randomly select another module from the module database, provide its source code to the LLM, and instruct it to incorporate the key ideas of that module into the target module. Furthermore, since mutating only one module at a time can slow down the evolutionary process, we also incorporate the repeat previous operation strategy, following LLMasTool. This setup leverages the module database in the same way as LLMasTool.

Agentic Generation: We also introduce an agentic generation approach, in which an agentic framework equipped with an agent harness is given access to the source code used to construct the module database. This allows the agent to freely explore and reference the available source code while improving the architecture.

We evaluate these approaches using three LLMs: Qwen3-8B, which was used in the original paper; Qwen3.8-27B, which supports tool calling; and GPT-6.1-Terra. For the agentic harness, we use GitHub Copilot CLI version 1.0.112.

The results are summarized in the following table. Validity rate represents the proportion of generated architectures that pass the feasibility check and successfully run. #Tokens denotes the average number of tokens (input + output) required to generate one valid, executable architecture, including tokens consumed by unsuccessful attempts.

LLMframework#TokensValidity RateAccuracy
Qwen3-8BKnowledge-based Generation71,00012.828.62
Qwen3-8BModule Generation45,00015.129.50
Qwen3-8BLLMasTool48,00023.440.78
Qwen3.8-27BKnowledge-based Generation45,00020.130.59
Qwen3.8-27BModule Generation24,00026.131.78
Qwen3.8-27BAgentic Generation2,600,00032.234.34
Qwen3.8-27BLLMasTool18,00053.339.89
GPT6.1-TerraAgentic Generation682,00078.132.42

First, as also demonstrated in Table 2 of the main paper, knowledge-based generation fails to achieve competitive performance. Second, although module generation leverages the module database in the same way as LLMasTool, it still underperforms LLMasTool. This suggests that, rather than asking the LLM to generate new modules from scratch, it is more effective to compose existing modules that have been previously designed by human experts. In our observations, module generation tends to produce conventional and commonplace modules and architectures, which limits its ability to achieve substantial performance improvements over existing architectures.

AuthorsOpenReview ↗

Response to Reviewer rjWE Part 2

Finally, for agentic generation, we employ the latest Copilot harness and the highly capable Qwen3.8-27B and GPT-6.1-Terra, yet the resulting performance remains limited. We believe that the discrepancy between the success of agentic frameworks in agentic research and their limited effectiveness in NAS stems from the fundamental difference in task difficulty and evaluation criteria. In agentic research, a key challenge is whether an agent can construct a pipeline from scratch that successfully solves a given task. Indeed, benchmarks such as Kaggle competitions often evaluate agents based on whether they can achieve a relatively competitive ranking, such as placing in the top 10%. In contrast, NAS requires improving upon architectures for tasks that have already been extensively optimized by human experts. Consequently, achieving state-of-the-art performance requires going beyond what conventional agentic frameworks can offer. We argue that coarse-to-fine decision making in LLMasTool is necessary to systematically explore architectural spaces beyond the scope of the LLM's prior knowledge.

Reliability of Architecture Generation and Search Efficiency

We appreciate this important point. First, we clarify that the numbers of explored architectures (e.g., 100 and 500) reported in our experiments refer to the number of architectures that successfully passed the feasibility checks, including computational cost and execution validity checks.

Additionally, we evaluated the validity rate (the proportion of generated architectures that pass the feasibility checks) and search efficiency. The results are reported in the table in the previous section.

Our results show that the validity rate generally improves when using more capable LLMs, with agentic generation achieving particularly high validity rates. We conjecture that this is because agentic frameworks already perform various checks and iterative trial-and-error procedures internally. Nevertheless, LLMasTool achieves a higher validity rate than direct LLM-based code generation without such checking or debugging mechanisms.

On the other hand, agentic frameworks consume a substantially larger number of tokens due to their internal debugging and iterative refinement processes, whereas LLMasTool is more computationally efficient because it uses the LLM only for localized design decisions.

The observation that less capable LLMs generally yield better performance, regardless of the framework used, remains an open question that requires further investigation and is left for future work. Nevertheless, our results consistently demonstrate that, when using the same LLM, LLMasTool can generate architectures with substantially lower LLM inference costs.

Stability of Planning Strategies

We agree that NAS results can exhibit substantial variance due to stochastic architecture generation and evaluation. To address this concern, we conducted additional experiments with three independent random seeds for each planning strategy.

planningImageNet-100 Accuracy
LLM's idea33.32 ± 0.31
c-to-f with uniform sampling38.79 ± 0.81
c-to-f with Thompson sampling33.31 ± 0.49
c-to-f with H-Thompson sampling40.18 ± 1.21

As demonstrated by these results, despite the relatively large variance, coarse-to-fine decision-making with localized LLM planning (c-to-f) substantially improves performance over planning driven solely by the LLM. Furthermore, the sampling strategy designed to introduce diversity into the planning process also plays an important role: H-Thompson sampling outperforms both uniform sampling and standard Thompson sampling. By hierarchically categorizing the broad search space, H-Thompson sampling effectively exploits prior experimental results while maintaining diversity in the exploration process.

Some of my concerns are addressed, but the fairness is still not very strong. I tend to keep my score.

Dear Reviewer rjWE,

We sincerely thank you for your time, effort, and valuable feedback during the conference review process, and we are glad to know that our rebuttal addresses most of your concerns and enhanced the paper's overall clarity.

We have revised the paper and compiled a summary of the improvements and additional experiments conducted in response to the comments from all reviewers. We would greatly appreciate it if you could take a moment to review these changes before making your final decision.

We understand that the fact that recent successful agentic frameworks and more capable LLMs do not outperform LLMasTool may raise concerns about the fairness of our experimental settings. We ourselves were both relieved and somewhat surprised by the results of our additional experiments.

Nevertheless, we have made every effort to ensure that the experimental settings are as fair as possible. We would sincerely appreciate it if you could point out any specific aspects that you consider potentially unfair.

Agentic frameworks and more capable LLMs are not failing; they have successfully identified models that improve upon the base model. However, we believe that further improving architectures that have already been extensively tuned by multiple researchers over a long period requires more than leveraging the prior knowledge of powerful LLMs. Rather, we believe that LLMasTool's approach, which actively explores beyond established prior knowledge and tries unconventional architectural modifications, is particularly effective in this challenging setting.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision based on Reviewer's Comments

Dear Reviewers,

Thank you very much for your constructive comments and valuable suggestions.

We have incorporated the responses provided during the rebuttal into the revised manuscript. We genuinely feel that your feedback has significantly improved the overall quality of our paper. Although the discussion period is now drawing to a close, we would greatly appreciate it if you could take another look at the revised manuscript before making your final decisions.

The main changes are summarized below.

  • Evaluation with Agentic Frameworks and Generally Stronger LLMs

    We believe that demonstrating the limitations of agentic frameworks and stronger LLMs is particularly important. These results highlight the importance of LLMasTool's coarse-to-fine decision-making strategy, which enables exploration beyond the scope of an LLM's prior knowledge. This suggests that our approach remains valuable even as LLMs continue to become more capable. We therefore added these evaluations to the main paper.

  • Evaluation under DARTS Settings

    We agree that evaluating the proposed method under multiple benchmark settings is important for demonstrating its generality. We have added the corresponding experiments in Appendix C.9.

  • Evaluation on ImageNet-1K

    We agree that investigating scalability is essential. We have therefore added the ImageNet-1K evaluation in Appendix C.8. Experiments under settings aligned with other NAS methods will be included in the camera-ready version.

  • Evaluation of the Applicability of Zero-Shot NAS Metrics

    We evaluated existing zero-shot NAS metrics and found that they do not perform effectively in open-ended search spaces such as the one explored by LLMasTool. We have added these results to Appendix C.11. We believe this finding provides valuable insights into potential directions for future research.

  • Other Improvements

    We have made several additional minor revisions throughout the paper to improve its clarity and readability.

  • Evaluation of Variance across Multiple Runs

    We have added standard errors to the 500-trial results in Table 1, demonstrating that LLMasTool consistently achieves strong performance. The standard errors for the ablation study in Table 2 are currently being measured and will be included in the camera-ready version.

Once again, we sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We hope that the revisions adequately address your concerns, and we would greatly appreciate your reconsideration of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.