- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article proposes the LLMasTool neural architecture search framework, which no longer allows large models to directly generate complete networks, but instead uses LLM as a local decision-making tool. The method combines source code module mining, hierarchical architecture tree, and coarse to fine evolution strategy, relying on predefined tree transformation operations to modify the structure, and using hierarchical Thompson sampling selection operations and prompt categories. Compared to NADER, we achieved stable accuracy improvement on the CIFAR and ImageNet datasets under a 500 search budget; The paper is accompanied by various components such as rich ablation, validation tree transformation, module library, sampling hierarchy, model size, and training rounds.
Critical strengths
The paper conducted a complete and controllable component ablation experiment with solid experimental evidence. The results show that tree structure editing is superior to direct code generation, and the mined reusable module library has significant effects. Hierarchical Thompson sampling has a positive effect on search; At the same time, stability was verified under different LLM parameter quantities and longer training cycles. The entire system does bring observable performance improvements, and its practical value cannot be denied just because it is built on existing technology.
Critical weaknesses
There are two core weaknesses. Firstly, the paper focuses on the discovery of an open architecture that breaks through the manual search space. However, the actual search space is still limited by fixed module libraries and preset tree operations, and there is no evidence to suggest that it can generate new computing primitives outside the module library or rewrite the internal computing logic of the module; The experiment can only prove that module combination reuse has good effects, and it is difficult to support the proposition of "open discovery". Secondly, there is a lack of controlled strong agent comparison: only different sizes of Qwen3 were tested within the same protocol, without comparison with other large model agents such as GPT and DeepSeek under the same budget and module conditions. Therefore, the current advantage cannot be extended to other LLM systems.
Questions for authors
The main motivation for the paper is to break through the manual search space limitations of traditional NAS, but LLMasTool still relies on pre built module databases and fixed tree transformation operations. Although this approach can increase the flexibility of combining existing modules, its essential difference from traditional module level NAS is not yet clear.
Please further explain: What new architectural structures can this method explore beyond existing module combinations and parameter adjustments? Is there any specific experiment that can prove its ability to discover new computing structures not included in the module library? If the search mainly relies on the recombination of existing modules, how does the author define the specific meaning and applicable boundaries of * open-ended NAS * in the paper?
The paper tested Qwen3 models of different scales in the appendix, but has not yet compared other model families and stronger agent-based architecture search methods under unified experimental conditions. Considering that the core viewpoint of the paper is to use LLM as a local decision-making tool, rather than allowing it to autonomously generate a complete network, this comparison is more important for verifying the advantages of the method.
Can the author explain whether the complete Agent method using GPT, DeepSeek, and other models is still significantly weaker than LLMasTool under the same search budget, module resources, execution checks, and feedback mechanisms? Can the existing results distinguish the benefits brought by structured search strategies from the impact caused by differences in underlying LLM capabilities?
Table 3 shows that using mining module libraries can bring significant performance improvements compared to basic module libraries, while Repeat Previous and Hierarchical Thompson Sampling in Table 2 also have a significant impact. Therefore, the current results cannot fully distinguish the independent contributions of module resources, architecture representation, and search strategies.
Can the author further explain how much independent benefit does Hierarchical Thompson Sampling have compared to other conventional search strategies, given fixed module databases, tree transformation operations, and search budgets? In addition, can the main results of 500 searches in Table 1 provide the mean and standard deviation of multiple independent runs to verify the stability of performance improvement?
6 comments
Response to Reviewer FMMt Part 2
Comparison Across LLM Families and Fully Autonomous Agent
We appreciate the reviewer’s concern that evaluating different Qwen3 model sizes alone is insufficient to establish the merits of LLMasTool against powerful LLM and agentic frameworks.
To investigate this, we conducted additional experiments using different LLM, including closed LLM of GPT6.1-Terra and newer open LLM of Qwen3.8-27B. We used Github Copilot CLI version 1.0.112 for a agentic harness. The results are summarized below. Validity rate represents the proportion of generated architectures that pass the feasibility check and successfully run. #Tokens denotes the average number of tokens (input + output) required to generate one valid, executable architecture, including tokens consumed by unsuccessful attempts.
| LLM | evo. step | Framework/Harness | #Tokens | Validity Rate | Accuracy |
|---|---|---|---|---|---|
| Qwen3-8B | code-gen | - | 71,000 | 12.8 | 28.62 |
| Qwen3-8B | tree-trans | LLMasTool | 48,000 | 23.4 | 40.78 |
| Qwen3.8-27B | code-gen | - | 45,000 | 20.1 | 30.59 |
| Qwen3.8-27B | code-gen | Copilot | 2,600,000 | 32.2 | 34.34 |
| Qwen3.8-27B | tree-trans | LLMasTool | 18,000 | 53.3 | 39.89 |
| GPT6.1-Terra | code-gen | Copilot | 682,000 | 78.1 | 32.42 |
Interestingly, NAS performance tends to deteriorate when using LLMs that are generally considered more capable. Investigating the underlying reasons is left for future work. As discussed in Appendix C.5, one possible hypothesis is that more capable LLMs may exhibit stronger prior biases, potentially limiting their ability to explore diverse trials. For this reason, simply using a more capable LLM or an agentic harness cannot serve as a substitute for LLMasTool.
Furthermore, when using the same underlying LLM, LLMasTool substantially outperforms both direct LLM-based NAS and agentic harnesses such as Copilot, demonstrating the effectiveness of our proposed framework.
Response to Reviewer FMMt Part 3
Independent Contributions of Architecture Representation, Module Database, and Search Strategy
We conducted an ablation study in which the main proposed components were incrementally added on ImageNet-100 dataset.
| representation | module | planning | repeat | sampling | accuracy |
|---|---|---|---|---|---|
| code | - | - | - | - | 28.62 |
| tree | primitive | LLM's idea | w/o | uniform | 30.24 |
| tree | mined | LLM's idea | w/o | uniform | 33.45 |
| tree | mined | c-to-f | w/o | uniform | 36.18 |
| tree | mined | c-to-f | w/ | uniform | 38.86 |
| tree | mined | c-to-f | w/ | H-Thompson | 40.78 |
As the results show, each of the proposed components contributes positively to the overall performance.
Unfortunately, we did not have sufficient time to repeat all ablation experiments multiple times and report the standard errors. However, we plan to conduct these additional experiments before the camera-ready submission to provide more reliable results. For some ablation studies investigating the effectiveness of the sampling strategies, we have already completed three independent runs. Please refer to our response to Reviewer rjWE (Stability of Planning Strategies) for these results.
Statistical Significance and Stability Across Independent Runs
We appreciate the reviewer’s suggestion to report the mean and standard deviation of the main results of 500 searches. The results of three runs are reported below.
| Method | CIFAR-10 val | CIFAR-10 test | CIFAR-100 val | CIFAR-100 test | ImageNet16-120 val | ImageNet16-120 test |
|---|---|---|---|---|---|---|
| NADER | 91.55 | 94.62 | 75.72 | 76.00 | 50.20 | 50.52 |
| LLMasTool | 93.00 ± 0.18 | 95.20 ± 0.18 | 77.57 ± 0.25 | 77.82 ± 0.15 | 52.39 ± 0.20 | 53.18 ± 0.48 |
With 500 search trials, LLMasTool achieves even more stable performance than with 100 trials. These results show that LLMasTool consistently outperforms previous methods.
Response to Reviewer FMMt Part 1
We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the reviewer’s recognition of the effectiveness of our component ablations, the benefits of the mined module library, and the positive contributions of hierarchical Thompson sampling. We agree that the scope of open-ended NAS, the generality across different LLM backbones, and the independent contributions of individual components deserve further clarification. We address these concerns below.
Clarification of Open-Ended NAS and the Difference from Conventional Module-Level NAS
We agree that the distinction between LLMasTool and conventional module-level NAS needs further clarification. In LLMasTool, open-endedness refers to the ability to expand the architectural search space beyond manually predefined architecture templates through automatically mined modules and hierarchical structural compositions. LLMasTool can automatically construct a module database from arbitrary PyTorch source code. Furthermore, by leveraging the LLM's ability to understand source code, it can automatically identify valid module combinations, as well as tunable hyperparameters and their feasible ranges. As a result, the combinations of modules and hyperparameter configurations give rise to an almost infinite, open-ended search space.
Conventional NAS search spaces tend to become outdated over time as new architectural designs emerge. LLMasTool addresses this limitation by allowing newly proposed modules from other research papers to be incorporated simply by adding their source code to the module database. This enables LLMasTool to continuously propose architectures that incorporate the latest advances in the field without requiring manual redesign of the search space. We believe this is one of the major advantages of LLMasTool over existing NAS methods.
Although the current framework does not support the proposal of low-level computational primitives, such as custom CUDA kernels, its "create module operation" enables the creation of new modules, similar to those designed by researchers and engineers. In fact, LLMasTool has already discovered several interesting module designs that, to the best of our knowledge, have not been reported in previous studies.
For example, Lines 2782–2827 of the architecture discovered for CIFAR-100 reveal a module immediately following the stem layer in which an Attention branch and a CNN branch are processed in parallel, and their outputs are merged through element-wise multiplication, as shown below.
Interestingly, similar structures are also found in the architectures discovered for CIFAR-10 and ImageNet16-120. This suggests that such parallel Attention-CNN modules may be particularly effective when placed immediately after the stem layer. We believe this observation could provide useful insights for future architecture design. Furthermore, the best architecture discovered on ImageNet16-120 incorporates an even more complex module, demonstrating LLMasTool's ability to discover better module structures.