- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article proposes EvoTreeNAD, a lineage guided LLM neural architecture discovery method. The architecture and its evolutionary descendants are constructed into a lineage tree, which relies on the Top percentile family value to guide the selection of evolutionary branches and explore potential branches. The paper conducted experiments on CIFAR-10/100 and MedMNIST-v2, including multiple rounds of independent search, multiple training seeds, failure rate and cost statistics; Under a fixed OSS20B and 100 search budgets, this strategy outperforms baselines such as greed and family mean. At the same time, two relevant theoretical theorems are presented in the article to characterize the statistical properties of the upper tail of the reward spectrum. There are three core questions raised in the review: the lack of comparison between Top percentile lineage routing and mature tree searches such as UCT/MCTS, which can easily degrade to maximum value selection in small family sizes; The main result (GPT-4.1, 300 search rounds) is inconsistent with the protocol of controlled validation in Table 2, making it difficult to transfer conclusions; The setting of modifiable rules and loss functions results in attribution ambiguity, and the theoretical asymptotic conclusion has limited explanatory power for limited search budget scenarios.
Critical strengths
This work proposes an evolutionary paradigm of lineage persistence storage and value routing based on family branches, which can trace back and continue to develop historical branches with clear search logic. The experimental system is relatively solid: covering multiple datasets for image classification, reporting multiple independent searches, training seeds, code generation failure rates, and computational costs; Under the control setting of unified OSS20B and 100 trials, repeated experiments consistently outperformed baselines such as greed and family mean, with reproducible empirical support. The analysis of matching theory is not simply formalized in words, but provides mathematical conclusions on the distribution of family rewards, enriching the analytical perspective of search mechanisms.
Critical weaknesses
The independent innovation boundary of the Top penalty family routing is unclear, and there is a lack of comparison with classic tree search strategies such as UCT/MCTS in the same framework; When the family size is small, this mechanism is prone to degenerate into max selection, and the unique gain under limited budget is not quantified.
Inconsistent experimental protocols: The controlled ablation in Table 2 is all based on OSS20B with 100 searches, while the best result in the paper uses GPT-4.1 with 300 searches and continues the existing search trajectory, which cannot directly prove that this lineage selection strategy has equal advantages under the main result setting.
There is ambiguity in performance attribution: the evaluation rule is to pre select multiple candidates and select the best report, which can easily lean towards optimism; At the same time, Idea Agent allows modification of the loss function, but cannot guarantee that the performance improvement comes entirely from the architecture itself; In addition, the theoretical conclusion is asymptotic, and its guiding and explanatory role for limited search budget scenarios such as 100-300 times has not been fully demonstrated.
Questions for authors
1、How does the proposed highest percentile family routing compare to UCT/MCTS under the same search protocol and budget? Will it bring sustained benefits when the family size is small and the mechanism is reduced to its maximum selection?
2、The controlled ablation in Table 2 was tested 100 times using OSS20B, while the best reported results were tested 300 times using GPT-4.1 with continuous trajectory. Can you verify if the advantage of your family tree selection still exists under the GPT-4.1 300 experimental setup?
4 comments
Response to Reviewer WGk8 (2/2)
3. Theoretical interpretation of routing behavior
For a fixed family size, averaging multiple top rewards reduces the worst-case influence of any single reward compared with using only the maximum (Appendix A.2, "Properties of Family-Value Rules", Proposition 2). Maintaining a top-percentile value a fixed amount above a reward threshold requires a nonvanishing fraction of high-reward family members as the family grows. Isolated early successes therefore cannot sustain that gap indefinitely, and weak descendants can change branch rankings even when the family maximum is unchanged. Under its variation and convergence assumptions, Theorem 2 gives an exact relation between limiting top-percentile family values and long-run high-reward production.
4. Architecture design and performance attribution
Section 3.1 ("Task Setting") defines an architecture as a complete trainable model specification, including its model-defined loss. Within each controlled comparison, all routing variants have the same freedom to design the network and its loss, and use the same LLM agents, task interface, evaluator, and external training protocols (Appendix D.3, "Full-Fidelity Evaluation Protocol", Table 8). The matched CIFAR-10 routing comparisons show that top-percentile routing discovers higher-performing trainable architectures than the alternative routing rules.
We will add the UCT variant to Table 2. The GPT-4.1/300-iteration routing comparisons and branch-switch analysis will be included in the appendix and referenced from the main text.
I think my concerns are resolved and I raise my rating.
Thank you for your thoughtful review and constructive feedback. We are glad that our clarifications and additional experiments have addressed your concerns, and we sincerely appreciate your time, engagement, and positive reassessment of our work! Thank you so much!
Response to Reviewer WGk8 (1/2)
We thank the reviewer for recognizing the genealogy-guided search design, the evidence across tasks and discovery runs, and the mathematical analysis. EvoTreeNAD outperforms UCT in the 100-iteration comparison. With GPT-4.1/OSS20B and 300 iterations, it also outperforms greedy continuation, full-family mean, and UCT. The routing traces include switches to branches with lower family maxima.
1. UCT comparison and small-family behavior
UCT selects children using empirical mean rewards augmented by a visit-dependent exploration bonus, whereas EvoTreeNAD uses top-percentile family values for lineage routing without an explicit exploration bonus.
We added three independent UCT runs (Kocsis and Szepesvari, 2006) with the same OSS20B agents, 100-iteration budget, discovery split, and evaluation protocol. Three candidates per run were preselected by discovery reward. The table reports the highest full-fidelity test accuracy among them.
Controlled CIFAR-10 comparison of EvoTreeNAD and UCT:
Under the matched 100-iteration protocol, EvoTreeNAD achieves 97.25% mean accuracy versus 96.94% for the exploration-oriented UCT variant.
A family includes the node and all its descendants (Section 3.2, "Architecture Genealogy and Family Values"). With τ=0.1, the top-percentile family value equals the maximum for families of at most ten members, then averages additional rewards as the family grows. Under maximum-family routing, the winning child remains selected at a saturated parent (Proposition 1). Under top-percentile routing, additional descendant rewards can change the child ranking.
The table shows two routing decisions from a 300-iteration EvoTreeNAD run using GPT-4.1/OSS20B and τ=0.2. Before each decision, the previously followed family grew from five to six members, bringing a second reward into its top-percentile average. All values are measured at the corresponding decision.
Branch switches under top-percentile routing:
At both decisions, the newly selected branch has a higher top-percentile family value despite a lower family maximum. Top-percentile routing therefore redirects development where maximum-family routing would retain the previous branch. The first switch occurs at iteration 34, showing that this distinction can emerge early in discovery.
2. Routing under GPT-4.1/OSS20B and 300 iterations
We compare the existing EvoTreeNAD run using τ=0.1 with four new 300-iteration discovery runs, one each for greedy continuation, full-family mean, UCT, and top-percentile routing with τ=0.2. The new runs start from empty roots. All five use GPT-4.1 as the Idea Agent and OSS20B as the Code Agent, with the same discovery split and evaluator.
The Reward-Top1 comparison selects the architecture with the highest discovery reward from each run. The table reports its best test accuracy during full-fidelity training and its test accuracy at epoch 500.
CIFAR-10 results with Reward-Top1 selection (GPT-4.1/OSS20B, 300 iterations):
With architecture selection based on discovery reward and evaluation fixed at epoch 500, EvoTreeNAD outperforms greedy continuation, full-family mean, and UCT. The advantage persists under the agent configuration and budget used for the main results.
Under candidate-set reporting, we select multiple candidates from each discovery run by discovery reward and evaluate them under the full-fidelity protocol. We report the best test accuracy among these candidates.
CIFAR-10 results with candidate-set reporting (GPT-4.1/OSS20B, 300 iterations):
EvoTreeNAD maintains its advantage over the three routing controls under both Reward-Top1 selection and candidate-set reporting, including at epoch 500.