- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an autonomous research team in which 16 instances of the same LLM receive different disciplinary profiles, retain private longitudinal histories, share experimental evidence, and are selected by a discounted-UCB scheduler. Under an 80-iteration, FLOP-matched search over approximately 150M-parameter linear-attention models, the team discovers Context-First DeltaNet (CFDN), which is then evaluated at 1.3B parameters against GDN, KDA, and GDN2; the paper reports that disciplinary profiles, private memory, and adaptive allocation each improve the best discovered utility.
Critical strengths
- The paper gives a concrete end-to-end protocol connecting hypothesis generation, review, implementation, training, evaluation, reflection, memory, and adaptive allocation.
- The method and experimental protocol are unusually explicit about researcher profiles, shared versus private information, FLOP allocation, speed checks, search feedback, and the discovered recurrence.
- The team produces a tangible architecture discovery and evaluates it at both search scale and 1.3B scale across language modeling, reasoning, synthetic retrieval, and real-world retrieval.
- The organizational ablations and the LM-only objective comparison are useful attempts to separate the roles of researcher organization and search objective.
Critical weaknesses
- All researcher, reviewer, and implementation roles use the same proprietary GPT-6 Astra family, only one of the 16 profiles is shown, and the paper does not report complete agent artifacts or total GPU, wall-clock, and API cost.
- I still think the six-step proposal presented yesterday is clear and straightforward. However, it seems the analysis section lacks sufficient ablation on the contribution of these steps.
For instance:
-
Do proposals differ across models, and does providing different contexts lead to noticeable gaps in the generated proposals?
-
To what extent do the proposal, implementation, and reflection steps individually contribute to the final accuracy of the policy?
-
Neural Architecture Discovery via Autonomous Evolution, Can you describe in a single sentence the additional core issue or problem covered in this article compared to the background of the previous one?
Questions for authors
see above
3 comments
Thank you for reporting the GPU hours and API costs and for showing how specific researcher profiles informed the proposals. The distinction from ASI-Arch is clearer now. I understand why the backbone and within-cycle stages were held fixed: the ablations compare team organizations under a shared workflow, rather than isolating the contribution of each research stage. If the Sol runs become available during discussion, I’d be most interested in whether the relative differences among the organizational variants persist on the new backbone.
Follow-up on GPT-6.1 Sol results.
We have now completed the comparison between UCB and uniform researcher selection using GPT-6.1 Sol at high reasoning effort. We ran each method for 50 cycles. Both use the same expert profiles, persistent researcher memory, search objectives, and per-candidate training budget.
The following table reports the performance of the architecture with the highest search utility from each run, together with their final evaluation results.
| Researcher selection | Raw search utility ↑ | LM PPL ↓ | CSR without overlap ↑ | NIAH without overlap ↑ |
|---|---|---|---|---|
| UCB (ours) | 0.9798 | 11.8768 | 43.16% | 28.51% |
| Uniform selection | 0.8410 | 12.1939 | 42.43% | 30.31% |
With GPT-6.1 Sol, UCB achieved a higher final search utility, demonstrating its advantages in discovery. In the final evaluation, the architecture found by UCB also achieved a lower language modeling perplexity and a higher commonsense reasoning accuracy. Uniform selection achieved a higher NIAH accuracy.
We wonder if these results address your concern about the robustness across backbones, and if so, would you consider updating your assessment?
We thank the reviewer for the valuable comments. We address each point below.
W1: Model choice, profiles, and experimental cost.
We used GPT-6 Astra at high reasoning effort across experiments to hold the underlying model fixed when comparing team organizations. To test how well our findings transfer across models, we added additional experiments to repeat the 150M-scale organizational comparisons with GPT-6.1 Sol at high reasoning effort. Given the available time and compute, these runs are still in progress, and we will report the results as soon as they are available.
We will include all 16 researcher profiles and the role prompts in the revision. The following table reports the total GPU hours and API cost for each of the compared variants.
As can be observed, all variants spend similar GPU hours, since they run for the same number of cycles and have a controlled training budget in each cycle. For API cost, the fresh memory variant has the lowest cost because agents in each cycle have a fresh session and thus a shorter context. The remaining three variants have similar API costs.
W2: Contributions of individual research stages.
We agree that ablating individual stages would help establish how each contributes to discovery. However, our focus in this paper is how to organize a research team under a fixed experimental budget. We therefore held the stages within each cycle fixed while varying researcher profiles, private memory, and researcher selection. We leave to future work whether independent review improves candidate quality, whether a dedicated implementer improves outcomes compared with having the proposing researcher implement its own idea, and whether post-experiment reflection improves subsequent discoveries.
Q1: How do models and researcher profiles affect proposals?
Each profile gives agents a different set of technical concepts, references, and reasoning patterns. Along CFDN's lineage, this guidance shows up in the mechanisms researchers proposed, as described in the following table.
These examples show how disciplinary profiles guide concrete design decisions during the discovery.
Q2: How do individual steps of proposal, implementation, and reflection contribute to the final performance?
Please see our response to W2.
Q3: What additional question does this paper address beyond ASI-Arch?
Building on ASI-Arch's demonstration that agents can autonomously propose, review, implement architectures, and analyze experimental results, we study how to improve discovery under a fixed experimental budget by organizing a team of agents with diverse expertise, enabling agents to build on one another's findings, and adaptively allocating experiments across the team.