acceptodds
Under review as a conference paper at ICLR 2027

Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

Abstract

Autonomous agents can increasingly propose ideas, modify code, and run experiments, creating an opportunity to accelerate scientific discovery using teams of collaborative AI researchers. To escape local minima in the design space induced by common reasoning patterns, we propose to encourage orthogonal exploration by constructing the team using agents with varying expertise. However, realizing this design requires more than diverse backgrounds. When experiments are costly, a team can test only a limited number of ideas, thereby creating the need to balance exploration and exploitation when selecting a subset of researchers to propose and execute new experiments. Furthermore, each result should shape what the team tries next. The team must decide where to deepen an investigation, when to explore alternatives, and how to carry lessons across researchers. We therefore ask how AI researchers with different scientific backgrounds can build on one another's findings and sustain progress under a limited experimental budget. To address these challenges, we build a team of persistent AI researchers, each grounded in a different discipline, who keep private research histories, share experimental evidence, and receive research opportunities through adaptive allocation. We demonstrate the effectiveness of this design by deploying the team to search for linear attention architectures at 150M parameters. We then scale the team's strongest discovery to 1.3B parameters, where it outperforms GDN2 on language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks. In controlled comparisons, removing any single organizational component from our design yields markedly weaker discoveries: without disciplinary backgrounds, the search plateaus early; without private histories, researchers make less progress despite identical shared evidence; and uniform selection falls short of adaptive allocation within the same budget. Tracing the discovery shows researchers applying principles from their own fields to refine one another's ideas. These results suggest that sustained autonomous discovery depends not only on stronger agents, but also on who explores, what they remember, and how research effort is allocated.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Auto-research, linear attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DR6xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an autonomous research team in which 16 instances of the same LLM receive different disciplinary profiles, retain private longitudinal histories, share experimental evidence, and are selected by a discounted-UCB scheduler. Under an 80-iteration, FLOP-matched search over approximately 150M-parameter linear-attention models, the team discovers Context-First DeltaNet (CFDN), which is then evaluated at 1.3B parameters against GDN, KDA, and GDN2; the paper reports that disciplinary profiles, private memory, and adaptive allocation each improve the best discovered utility.

Critical strengths
  1. The paper gives a concrete end-to-end protocol connecting hypothesis generation, review, implementation, training, evaluation, reflection, memory, and adaptive allocation.
  2. The method and experimental protocol are unusually explicit about researcher profiles, shared versus private information, FLOP allocation, speed checks, search feedback, and the discovered recurrence.
  3. The team produces a tangible architecture discovery and evaluates it at both search scale and 1.3B scale across language modeling, reasoning, synthetic retrieval, and real-world retrieval.
  4. The organizational ablations and the LM-only objective comparison are useful attempts to separate the roles of researcher organization and search objective.
Critical weaknesses
  1. All researcher, reviewer, and implementation roles use the same proprietary GPT-6 Astra family, only one of the 16 profiles is shown, and the paper does not report complete agent artifacts or total GPU, wall-clock, and API cost.
  2. I still think the six-step proposal presented yesterday is clear and straightforward. However, it seems the analysis section lacks sufficient ablation on the contribution of these steps.

For instance:

  1. Do proposals differ across models, and does providing different contexts lead to noticeable gaps in the generated proposals?

  2. To what extent do the proposal, implementation, and reflection steps individually contribute to the final accuracy of the policy?

  3. Neural Architecture Discovery via Autonomous Evolution, Can you describe in a single sentence the additional core issue or problem covered in this article compared to the background of the previous one?

Questions for authors

see above

3 comments

We thank the reviewer for the valuable comments. We address each point below.

W1: Model choice, profiles, and experimental cost.

We used GPT-6 Astra at high reasoning effort across experiments to hold the underlying model fixed when comparing team organizations. To test how well our findings transfer across models, we added additional experiments to repeat the 150M-scale organizational comparisons with GPT-6.1 Sol at high reasoning effort. Given the available time and compute, these runs are still in progress, and we will report the results as soon as they are available.

We will include all 16 researcher profiles and the role prompts in the revision. The following table reports the total GPU hours and API cost for each of the compared variants.

VariantH100 GPU-hoursAPI cost (USD)
Full team with UCB1,793.71,902.85
No background1,789.81,744.68
Fresh memory1,744.81,429.68
Uniform selection1,867.81,808.52

As can be observed, all variants spend similar GPU hours, since they run for the same number of cycles and have a controlled training budget in each cycle. For API cost, the fresh memory variant has the lowest cost because agents in each cycle have a fresh session and thus a shorter context. The remaining three variants have similar API costs.

W2: Contributions of individual research stages.

We agree that ablating individual stages would help establish how each contributes to discovery. However, our focus in this paper is how to organize a research team under a fixed experimental budget. We therefore held the stages within each cycle fixed while varying researcher profiles, private memory, and researcher selection. We leave to future work whether independent review improves candidate quality, whether a dedicated implementer improves outcomes compared with having the proposing researcher implement its own idea, and whether post-experiment reflection improves subsequent discoveries.

Q1: How do models and researcher profiles affect proposals?

Each profile gives agents a different set of technical concepts, references, and reasoning patterns. Along CFDN's lineage, this guidance shows up in the mechanisms researchers proposed, as described in the following table.

CycleResearcher profileHow the profile informed the proposal
C21Linear attention and recurrent matrix memoryThe profile asks what information a fixed-size recurrent state should preserve. The researcher searched for affine delta rules and proposed a shared context vector alongside the address-dependent matrix, with both updated from the same prediction error.
C33Parallel and I/O-aware recurrent designThe profile emphasizes precomputable operations and efficient chunkwise training. The researcher used a static, learned query/key translation to change what the memory preserves under decay while keeping the recurrent state size fixed.
C44Bayesian filtering and innovationsThe profile emphasizes calibrating the strength of a correction to the scale of the evidence. The researcher drew on normalized LMS to compensate for the change in key energy introduced by C33's translation.
C68Predictive representations and information bottlenecksThe profile distinguishes immediate predictive accuracy from retaining information needed later. After C56 improved perplexity but hurt retrieval, the researcher proposed reversing the update order to test whether giving shared context priority would help future retrieval.

These examples show how disciplinary profiles guide concrete design decisions during the discovery.

Q2: How do individual steps of proposal, implementation, and reflection contribute to the final performance?

Please see our response to W2.

Q3: What additional question does this paper address beyond ASI-Arch?

Building on ASI-Arch's demonstration that agents can autonomously propose, review, implement architectures, and analyze experimental results, we study how to improve discovery under a fixed experimental budget by organizing a team of agents with diverse expertise, enabling agents to build on one another's findings, and adaptively allocating experiments across the team.

Thank you for reporting the GPU hours and API costs and for showing how specific researcher profiles informed the proposals. The distinction from ASI-Arch is clearer now. I understand why the backbone and within-cycle stages were held fixed: the ablations compare team organizations under a shared workflow, rather than isolating the contribution of each research stage. If the Sol runs become available during discussion, I’d be most interested in whether the relative differences among the organizational variants persist on the new backbone.

Follow-up on GPT-6.1 Sol results.

We have now completed the comparison between UCB and uniform researcher selection using GPT-6.1 Sol at high reasoning effort. We ran each method for 50 cycles. Both use the same expert profiles, persistent researcher memory, search objectives, and per-candidate training budget.

The following table reports the performance of the architecture with the highest search utility from each run, together with their final evaluation results.

Researcher selectionRaw search utility ↑LM PPL ↓CSR without overlap ↑NIAH without overlap ↑
UCB (ours)0.979811.876843.16%28.51%
Uniform selection0.841012.193942.43%30.31%

With GPT-6.1 Sol, UCB achieved a higher final search utility, demonstrating its advantages in discovery. In the final evaluation, the architecture found by UCB also achieved a lower language modeling perplexity and a higher commonsense reasoning accuracy. Uniform selection achieved a higher NIAH accuracy.

We wonder if these results address your concern about the robustness across backbones, and if so, would you consider updating your assessment?

Reviewer_MKitOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the organization of autonomous research teams for linear attention architecture discovery. The proposed framework combines disciplinary profiles, private histories, shared evidence, and adaptive researcher selection. It discovers Context-First DeltaNet and evaluates the architecture at 150M and 1.3B scales. The results are promising, although stronger search baselines and repeated runs are needed to establish the benefits of the team organization.

Critical strengths
  1. Meaningful motivation. Combining different research perspectives and allocating limited experimental opportunities are important problems for autonomous research.
  2. Coherent technical design. The framework connects researcher profiles and memory with proposal review, implementation, evaluation, and adaptive allocation. The reported discovery trajectory illustrates how researchers develop and refine one another's ideas.
  3. Substantial empirical validation. The paper includes organizational ablations and evaluates the discovered architecture under matched model-training FLOPs at two scales. It also evaluates real-world retrieval tasks that were not used during search.
Critical weaknesses
  1. Limited comparison with existing search methods. The experiments mainly compare internal variants and the resulting architectures. A comparable autonomous search baseline, such as an ASI-Arch-style workflow or a single persistent researcher, would help establish the value of the proposed team organization.
  2. Insufficient evidence of robustness. Each condition is represented by one reported search trajectory, and removing disciplinary profiles also removes substantial technical guidance. Repeated searches and an equally informative shared-profile control would help separate stable organizational benefits from search variability and additional prompt information.
  3. Incomplete separation of search feedback and final evaluation. Search repeatedly uses the same language-modeling benchmarks and subsets of the commonsense tasks subsequently used for full evaluation. The held-out real-world retrieval evaluation is valuable, but disjoint evaluation data and uncertainty estimates would strengthen the evidence of generalization beyond the search feedback.
Questions for authors
  1. How consistent are the organizational gains across independent search runs? A comparison with a single persistent researcher or an existing autonomous search workflow, under the same total budget and comparable technical guidance, would help establish the value of the team organization.
  2. Which final evaluation examples are disjoint from all search feedback, beyond the real-world retrieval suite already held out? Please report performance and uncertainty on these independent examples to clarify how much of the improvement generalizes beyond the repeatedly queried search benchmarks.
2 comments

We thank the reviewer for the valuable comments. We address each point below.

W1: Autonomous search baseline with comparable technical guidance.

As per the reviewer's suggestion, we added a single-agent baseline adapted from the workflows of ASI-Arch. Specifically, we retain its core autonomous research loop: proposal, review, implementation, training and evaluation, and reflection that informs later proposals; but we adapt it to our setting so that the comparison is consistent on what agents can modify, what feedback they receive, and how candidates are validated.

The main difference is how the discovery is organized. Our method uses 16 persistent researcher agents with distinct expert profiles and allocates cycles among them using UCB. The single-agent baseline assigns every cycle to one persistent researcher who can also consult all 16 profiles, so the baseline has access to the same technical guidance.

We matched the underlying model (GPT-6 Astra/high) and harness, starting architecture, search objectives, and experimental budgets. The following table shows the metrics during the search.

MethodBest cycleUtility ↑Search LM PPL ↓Search CSR ↑Search NIAH proxy ↑
Single-agent baseline710.64611.94144.43%41.93%
16-agent team with UCB681.04511.90244.63%50.00%

Within the same 80-cycle budget, our method found a higher-utility candidate, which supports the contribution of team organization under a common search protocol.

W2: Robustness across search runs.

We agree that one trajectory per condition leaves uncertainty about how consistently the results hold. We designed additional experiments to repeat the 150M ablations in Figure 3. Specifically, we repeat the experiments with GPT-6.1 Sol at high reasoning effort in place of GPT-6 Astra, which also tests the generalization across different models. Given the available time and compute, these runs are still in progress, and we will report the results as soon as they are available.

W3: Separation of search feedback and final evaluation.

We filter final evaluation examples by removing those that appear in the search process. For commonsense reasoning (CSR), we removed 2,051 examples, leaving 20,368 examples. For NIAH, we removed 768 examples and left 6,232 examples. The following table reports the macro averaging across eight CSR tasks and fourteen NIAH task-context combinations, as well as the 95% bootstrap confidence intervals.

ScaleModelCSR, full setCSR, no overlap [95% CI]NIAH, full setNIAH, no overlap [95% CI]
150MGDN42.8842.97 [42.00, 43.96]16.9416.94 [16.22, 17.68]
150MKDA41.9441.52 [40.57, 42.50]22.0322.20 [21.32, 23.10]
150MGDN243.6043.23 [42.28, 44.18]25.7925.86 [25.02, 26.69]
150MCFDN43.2943.00 [42.05, 43.96]29.7629.80 [28.96, 30.66]
1.3BGDN53.7353.37 [52.34, 54.40]54.9954.97 [54.03, 55.91]
1.3BKDA53.9053.55 [52.51, 54.60]59.1159.16 [58.45, 59.88]
1.3BGDN252.6551.92 [50.92, 52.95]66.8667.01 [66.15, 67.87]
1.3BCFDN54.5354.34 [53.30, 55.39]67.2967.24 [66.36, 68.13]

The nonoverlapping results show similar trends as the full-set results. At both scales, the CSR and NIAH rankings are unchanged. Together with the performance on real-world retrieval tasks, which do not occur in the search at all, they demonstrate that the improvements of the discovered architecture persist to some extent on unseen examples.

Follow-up on GPT-6.1 Sol results.

We have now completed the comparison between UCB and uniform researcher selection using GPT-6.1 Sol at high reasoning effort. We ran each method for 50 cycles. Both use the same expert profiles, persistent researcher memory, search objectives, and per-candidate training budget.

The following table reports the performance of the architecture with the highest search utility from each run, together with their final evaluation results.

Researcher selectionRaw search utility ↑LM PPL ↓CSR without overlap ↑NIAH without overlap ↑
UCB (ours)0.979811.876843.16%28.51%
Uniform selection0.841012.193942.43%30.31%

With GPT-6.1 Sol, UCB achieved a higher final search utility, demonstrating its advantages in discovery. In the final evaluation, the architecture found by UCB also achieved a lower language modeling perplexity and a higher commonsense reasoning accuracy. Uniform selection achieved a higher NIAH accuracy.

We wonder if these results and our initial response address your concerns, and if so, would you consider updating your assessment?

Reviewer_8C6mOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper uses autonomous research for architectural search of linear attention mechanisms. It proposes orthogonal explorations which helps escape local minima. They demonstrate their autonomous research method on improving linear attention architecture on a 150M scale then extending the found architecture to 1.5B.

Critical strengths

The method for selecting the researcher for the next iteration is correctly formulated as an exploration vs exploitation tradeoff and uses a formulated method as opposed to simply letting the agents decide by themselves which can skew into exploiting local minima.

Critical weaknesses

The main concern is that the evaluation in figure 3 which drives many of the contributions is a small-scale single run for each method on a small 150M parameter model. In addition, the metrics reported are on data that overlaps with the data given to the autoresearch as feedback which causes overfitting, GDN and KDA are general architectures that perform well across diverse tasks and scales while the autoresearch conducted in the paper overfits relatively few metrics on a fixed small set of tasks. Additionally, the experiments lack comparisons against previous autonomous-discovery methods like [1], [2], and from last years ICLR [3].

[1] Liu, Y., Nan, Y., Xu, W., Hu, X., Ye, L., Qin, Z., & Liu, P. (2025). Alphago moment for model architecture discovery. arXiv preprint arXiv:2507.18074.

[2] Pepe, A., Lin, C. Y., Magka, D., Acun, B., Foster, T. S., Wu, Y. N., ... & Bachrach, Y. (2026). Agentic discovery of neural architectures: AIRA-Compose and AIRA-Design. arXiv preprint arXiv:2605.15871.

[3] Lange, R., Imajuku, Y., & Cetin, E. (2026, April). Shinkaevolve: Towards open-ended and sample-efficient program evolution. In International Conference on Learning Representations (Vol. 2026, pp. 74026-74078).

Questions for authors

Can you incorporate and share results for the mentioned prior methods along with multiple runs and multiple scales. This would give a stronger scientific justification of the proposed method.

Can you separate the data exposed to the agent during research and what is reported.

These questions do not affect my recommendation:

“For all models, we adjust their training tokens to match the total FLOPs of training GDN: at 150M scale, we train GDN for 15B tokens; at 1.3B scale, we train GDN for 100B tokens. More expensive architectures therefore receive fewer training tokens.” How do more expensive architectures receive fewer training tokens if the 1.3B scale has 100B tokens compared to the 15B tokens that the 150M scale gets.

3 comments

We thank the reviewer for the valuable comments. We address each point below.

W1: Comparison with prior autonomous discovery methods.

We added a comparison against a baseline adapted from the workflows of ASI-Arch [1] and AIRA-Design [2]. A direct comparison with the original implementation is difficult because the systems differ in what agents can modify, what feedback they receive, and how candidates are validated. For example, ASI-Arch uses its own fitness function, which includes an LLM assessment of architectural quality, while AIRA-Design allows agents to edit the training script itself. In our setting, agents modify the architecture under a fixed training recipe, receive LM, CSR, and NIAH feedback, and must meet a training-time limit relative to the GDN2 baseline. Running each system under its native setting would mix differences in search organization with differences in the task and feedback.

We therefore adapted the baselines' core autonomous research loop to our setting: proposal, review, implementation, training and evaluation, and reflection that informs later proposals. The main difference is how the discovery is organized. Our method uses 16 persistent researcher agents with distinct expert profiles and allocates cycles among them using UCB. The single agent baseline assigns every cycle to one persistent researcher who can consult all 16 profiles (thus the baseline has access to the same external information). Both methods retain the same review, implementation, and reflection stages.

We matched the underlying model (GPT-6 Astra/high) and harness, starting architecture, search objectives, and experimental budgets. The following table shows the results.

MethodBest cycleUtility ↑Search LM PPL ↓Search CSR ↑Search NIAH proxy ↑
Single agent baseline710.64611.94144.43%41.93%
16-agent team with UCB681.04511.90244.63%50.00%

Within the same 80-cycle budget, our method found a higher-utility candidate, which supports the contribution of team organization under a common search protocol.

W2: Robustness across search runs.

We agree that one run per condition leaves uncertainty about how consistently the results hold. We designed additional experiments to repeat the 150M ablations in Figure 3. Specifically, we repeat the experiments with GPT-6.1 Sol at high reasoning effort in place of GPT-6 Astra, which also tests the generalization across different models. Given the available time and compute, these runs are still in progress, and we will report the results as soon as they are available.

W3: Overlap between search feedback and final evaluation.

We agree that it is important to measure the performance on data that is not used during the search process. To do that, we filter the final evaluation examples and remove those that occur in the search. For commonsense reasoning (CSR), we removed 2,051 examples, leaving 20,368 examples. For NIAH, we removed 768 examples and left 6,232 examples. The following table reports the macro averaging across eight CSR tasks and fourteen NIAH task-context combinations.

ScaleModelCSR, full set ↑CSR, no overlap ↑NIAH, full set ↑NIAH, no overlap ↑
150MGDN42.88%42.97%16.94%16.94%
150MKDA41.94%41.52%22.03%22.20%
150MGDN243.60%43.23%25.79%25.86%
150MCFDN43.29%43.00%29.76%29.80%
1.3BGDN53.73%53.37%54.99%54.97%
1.3BKDA53.90%53.55%59.11%59.16%
1.3BGDN252.65%51.92%66.86%67.01%
1.3BCFDN54.53%54.34%67.29%67.24%

The nonoverlapping results show similar trends as the full-set results. At both scales, the CSR and NIAH rankings are unchanged. Together with the performance on real-world retrieval tasks, which do not occur in the search at all, they demonstrate that the improvements of the discovered architecture do generalize to some extent to unseen examples.

Q: How are training tokens matched across scales?

We match training FLOPs within each scale. Training GDN for 15B tokens defines the 150M-scale's compute budget, and training GDN for 100B tokens defines a separate 1.3B-scale budget. Because CFDN requires more training FLOPs per token than GDN, it receives fewer tokens under each budget: 13.53B vs. 15.00B tokens at 150M and 87.17B vs. 100.00B tokens at 1.3B. We will make this within-scale comparison explicit in the revision.

Follow-up on GPT-6.1 Sol results.

We have now completed the comparison between UCB and uniform researcher selection using GPT-6.1 Sol at high reasoning effort. We ran each method for 50 cycles. Both use the same expert profiles, persistent researcher memory, search objectives, and per-candidate training budget.

The following table reports the performance of the architecture with the highest search utility from each run, together with their final evaluation results.

Researcher selectionRaw search utility ↑LM PPL ↓CSR without overlap ↑NIAH without overlap ↑
UCB (ours)0.979811.876843.16%28.51%
Uniform selection0.841012.193942.43%30.31%

With GPT-6.1 Sol, UCB achieved a higher final search utility, demonstrating its advantages in discovery. In the final evaluation, the architecture found by UCB also achieved a lower language modeling perplexity and a higher commonsense reasoning accuracy. Uniform selection achieved a higher NIAH accuracy.

We wonder if these results and our initial response address your concerns, and if so, would you consider updating your assessment?

I appreciate the new results with 6.1 Sol and the response to W3. Overall, I think UCB is an interesting concept which is a more rigorous way of performing autonomous scientific discovery and would like to see this project fleshed out more.

I will increase my score to a 2. I still feel like the comparison with prior research needs a more rigorous evaluation. And in general more results repeated across various setups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.