acceptodds
Desk rejected at ICLR 2027

AIM: Agentic Idea Management for Automated Research

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

27 views

Abstract

Frontier LLMs are increasingly used to automate scientific research through iterative search. We distinguish idea-driven search from solution-driven search and identify three core challenges: organizing evolving research ideas, selecting promising directions, and maintaining alignment between ideas and their implementations. To address these challenges, we introduce the Agentic Idea Manager (AIM), a fully autonomous framework for managing and exploring research directions in idea-driven automated research. Inspired by Bayesian optimization, AIM uses an Agentic Surrogate and an Agentic Acquisition mechanism to organize discovered ideas and guide their selection. A Solution Auditor maintains idea-solution integrity, while a Resource Planner adaptively allocates the remaining experimental budget across parallel search branches. Experiments on 10 AutoLab benchmark tasks show that AIM surpasses the strongest baseline by 1.6 percentage points on System Optimization tasks and 4.9 percentage points on long-horizon Model Development & CUDA tasks. Notably, AIM reaches the best baseline performance up to 3.1x faster in wall-clock time. We further provide a theoretical analysis of when searching over ideas becomes beneficial. Our analysis shows that explicit idea-level allocation makes semantic coverage directly controllable, and that broader coverage becomes increasingly valuable when competitive research directions are sparse among many plausible alternatives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Research, Automated Scientific Discovery, LLM Agents, Harness

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fKsfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the management and exploration of research ideas in automated research under limited experimental budgets. It proposes Agentic Idea Manager (AIM), a framework comprising four components: an Agentic Surrogate that dynamically organizes ideas into semantic clusters and ranks their promise; an Agentic Acquisition module that balances exploration and exploitation at both cluster and idea levels, dispatches candidates to parallel solvers, and generates new ideas from experimental feedback; a Solution Auditor that filters invalid results and corrects mismatches between ideas and their implementations; and a Resource Planner that adaptively allocates a fixed experimental budget across iterations. Across ten AutoLab tasks covering system optimization, model development, and CUDA programming, AIM improves average performance over the strongest baseline by 1.6 and 4.9 percentage points in the two task groups, respectively, and reaches baseline performance up to 3.1 times faster. The paper also provides a theoretical analysis explaining why broader semantic coverage can be valuable when competitive research directions are sparse.

Critical strengths

The paper identifies three challenges—organization, selection, and implementation consistency—giving the method’s motivation clear and concrete targets.

The average performance across both task groups exceeds that of ScientistOne, suggesting that the framework’s gains are not limited to a single task. The main text also acknowledges AdaEvolve’s advantage on the data-selection task.

Semantic Coverage provides an explicit definition of research-direction coverage, giving the discussion of search breadth a formal basis.

Critical weaknesses

The paper categorizes search methods as either idea-driven or solution-driven, but this distinction lacks strong justification. It is therefore unclear whether the taxonomy comprehensively covers all search methods.

Both the main component ablations and the more fine-grained surrogate ablations focus on Flash Attention, making it difficult to determine whether the same mechanisms explain the gains on model-development and CUDA tasks.

Table 5 shows that AIM uses substantially more input tokens than some baselines. While the common-goal comparison in Fig. 10 is valuable, the paper does not compare all methods under a unified total token budget.

Questions for authors

See Weakness

3 comments

We thank the reviewer for the constructive feedback and for recognizing the clarity of our motivation, the significance of our empirical findings, and the value of our theoretical discussion. Below, we address the reviewer's key concerns.


#1 Justification of the idea-driven vs. solution-driven categorization

Thank you for this important question! First of all, we would like to highlight other reviewers' comments about the categorization

"the distinction between solution-driven and idea-driven seems reasonable" -- Reviewer vDno

"Identifying ideation in contrast to validation&evaluation as an important search angle. Formal understanding between the two that provides useful insights of which strategy wins in what cases." -- Reviewer B2hq

Moreover, Peirce's account of scientific inquiry provides conceptual support for this distinction. He separates forming an explanatory hypothesis (retroduction), deriving its testable consequences (deduction), and assessing those consequences against experience (induction) [1]. In the automated research settings we study, the problem is already specified, and ideas are realized through executable implementations. Experimental feedback can therefore guide search at either level. Idea-driven search uses results to select ideas for subsequent implementation, while solution-driven search uses results to select and refine executable candidates.

Our categorization captures which of these two spaces serves as the primary search space. A method can explicitly search over ideas and delegate implementation to a solver or search directly over executable solutions. These possibilities encompass search over the idea and solution spaces defined in our formulation, including hybrid approaches. Therefore, Peirce's account supports the conceptual separation between these levels, while our categorization makes that separation operational for comparing automated research methods.

[1] Peirce, C. S. (1908). “A Neglected Argument for the Reality of God.”


#2 Ablations on Model-development & CUDA tasks

Great suggestion! We extend the ablation study to NTT Butterfly, a longer-horizon Model Development & CUDA task. We report the mean best score (%) and standard error across 3 independent runs. The table below shows the ablation study results on both Flash Attention and NTT Butterfly.

VariantFlash AttentionNTT Butterfly
AIM (full)90.5 ± 0.859.0 ± 1.3
(-) Agentic Surrogate85.9 ± 2.757.5 ± 2.0
(-) Agentic Acquisition89.2 ± 0.955.6 ± 0.5
(-) Solution Auditor87.6 ± 2.056.8 ± 1.6
(-) Resource Planner88.1 ± 0.756.7 ± 2.1

#3 Token cost and performance

AIM's input-token usage reflects its use of accumulated research context, including candidate ideas, evaluation history, and implementation lessons. This context allows the agent to compare research directions, revisit promising ideas, and use previous outcomes to guide subsequent experiments. The additional cost thus supports the agent's decisions about where and how to spend the experimental budget.

To make this trade-off more explicit, we will report AIM's score and cumulative token usage after the first iteration alongside the baselines' final results. The following comparison uses Flash Attention, with final-run values from Table 5.

MethodCheckpointBest score (%)Input tokens (M)Output tokens (M)Total tokens (M)Estimated cost (USD)
AIRA-EVOFinal76.30.502.202.70$27.40
ArborFinal77.214.400.2914.69$32.28
AIRA-MCTSFinal77.90.903.174.07$39.84
AIMIteration 183.618.980.4419.41$43.19
AIMFinal90.590.102.2592.35$207.20

After its first iteration, AIM achieves 83.6%, exceeding the final scores of Arbor (77.2%), AIRA-MCTS (77.9%), and AIRA-EVO (76.3%), at an estimated cost of 43.19 USD, comparable to their 27.40 - 39.84 USD. This comparison clarifies how much of AIM's advantage emerges at a comparable monetary cost. Its subsequent iterations spend additional tokens to refine the search and reach a final score of 90.5%.

However, we certainly agree that reducing this token overhead is an important direction for future work. In particular, instead of keeping all the accumulated research context from repeated rounds, we may investigate ways to compress experimental histories into concise evidence summaries, retrieving only the context relevant to each decision, and incrementally updating cluster assessments instead of repeatedly processing the entire idea pool. These changes could reduce input-token usage while retaining the accumulated evidence that informs AIM's research decisions.

Dear Reviewer fKsf,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Dear Reviewer fKsf,

Thank you again for your time and thoughtful feedback!

We understand this is a busy period, but with our discussion deadline coming to a close, we wanted to gently follow up. We have done our best to address your questions and would be happy to use the last minute of our discussion period to provide any further clarification. Could you please let us know if our responses addressed your concerns?

Thanks again, and we look forward to your response!

Best regards,

The Authors.

Reviewer_TwDvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AIM, a framework for managing research ideas under a limited experimental budget. AIM dynamically clusters and ranks ideas, selects candidates through exploration and exploitation, audits implementations to correct misattributed results, and adjusts branch allocation across iterations. On ten AutoLab tasks, AIM reports average normalized scores of 67.0 versus ScientistOne’s 65.4 on system optimization, and 55.8 versus 50.9 on model development and CUDA tasks. It also reaches ScientistOne’s final performance up to 3.1× faster on Flash Attention. The paper supplements these results with ablations, cost analyses, and theoretical arguments relating semantic coverage to discovering competitive research directions.

Critical strengths

Strengths:

  1. The paper identifies how mismatches between proposed ideas and actual implementations can contaminate subsequent research decisions. The examples involving exponential approximations and vectorization strategies clearly illustrate this failure mode.
  2. The evaluation includes multiple search frameworks, three independent runs, component ablations, and a solver substitution experiment. The appendix also reports runtime, execution counts, and token consumption, providing useful evidence beyond final scores.
Critical weaknesses
  1. The contributions of the new mechanisms are not that clear. Both the main and detailed ablations are restricted to Flash Attention, and controlled comparisons against random, greedy, UCB, or static clustering policies are missing.
  2. Each setting has only three runs, and approximately two-thirds of the aggregate gain over ScientistOne on model development and CUDA tasks comes from Data Select IFEval. All tasks come from AutoLab, with eight focused on systems or kernel optimization.
Questions for authors

See weaknesses, and:

Can you independently validate the Auditor and separately measure filtering and idea reconstruction under identical final evaluation rules, while preserving the distinction between the original hypothesis and the implemented method?

5 comments

We thank the reviewer for the constructive feedback and for recognizing the importance of the problem we address and the breadth of our empirical evaluation. We address the reviewer's key concerns below.


#1 Contribution clarification

Thank you for asking us to clarify our contributions! AIM's central contribution is to make idea management part of what the research agent does autonomously. As experimental evidence accumulates, the agent decides how to group and assess research ideas, which directions to pursue, and how to allocate its remaining budget. To our knowledge, AIM is the first automated research framework to jointly delegate the evolving organization of ideas and the exploration–exploitation decisions over that organization to the agent. The four components of AIM contribute distinct mechanisms:

  • Agentic Surrogate. AIM introduces an agent that repeatedly reorganizes the entire idea pool into semantic clusters using experimental evidence. Previous approaches organize ideas through predefined structures, such as refinement trees in Arbor or beam-style retention in ScientistOne.
  • Agentic Acquisition. The two-level exploration-exploitation mechanism allows the agent to adapt to the evolving organization of ideas and accumulated evidence. Previous approaches select candidates through greedy retention or predefined exploration heuristics.
  • Solution Auditor. Prior work, including ScientistOne, has established the importance of auditing and evidence traceability. AIM's contribution is to integrate auditing directly into the idea-management feedback loop, excluding invalid results from subsequent iterations to prevent misleading evidence from propagating into later decisions.
  • Resource Planner. Prior work such as AdaEvolve allocates resources using predefined scheduling rules. AIM delegates the allocation decision to the agent itself, autonomously balancing broader parallel exploration against additional rounds of sequential feedback.

Together, these components introduce specific mechanisms for autonomously organizing, selecting, and maintaining research ideas throughout the experimental process. Thus, each component adds something novel to the field of automated research.


#2 Ablation studies on different tasks

Thank you for the suggestion! We extend the ablation study to NTT Butterfly, a longer-horizon Model Development & CUDA task. We report the mean best score (%) and standard error across 3 independent runs. The table below shows the ablation study results on both Flash Attention and NTT Butterfly.

VariantFlash AttentionNTT Butterfly
AIM (full)90.5 ± 0.859.0 ± 1.3
(-) Agentic Surrogate85.9 ± 2.757.5 ± 2.0
(-) Agentic Acquisition89.2 ± 0.955.6 ± 0.5
(-) Solution Auditor87.6 ± 2.056.8 ± 1.6
(-) Resource Planner88.1 ± 0.756.7 ± 2.1

#3 Controlled baseline comparison

Our comparison with ScientistOne provides a system-level reference for greedy retention with random replenishment. ScientistOne retains the best-scoring ideas at each iteration and replenishes the pool with newly sampled ideas. Thus, the proposed random and greedy policies are already reflected in ScientistOne's search strategy.

UCB and static clustering are difficult to incorporate directly into AIM's evolving search space. In tree-based search, UCB attaches visit counts and rewards to persistent nodes. In AIM, cluster membership changes as ideas and evidence accumulate, requiring additional rules for transferring these statistics across iterations. Similarly, freezing the initial clusters requires rules for accommodating new ideas that do not fit any existing cluster. Consequently, implementing these policies would require changes to the idea-management framework beyond simply replacing the selection or clustering policy, making a clean controlled comparison difficult.

#4 Statistical Significance

We followed AutoLab's evaluation protocol, whose leaderboard also ranks methods by their average performance over three independent runs. To further quantify uncertainty, we analyzed aggregate performance across the five Model Development & CUDA tasks using an exact task-stratified paired bootstrap.

For each baseline, we resample matched run pairs with replacement within each task, compute the task-level AIM–baseline differences, and average these differences equally across tasks. We enumerate the bootstrap resamples with their corresponding probabilities and report the central 95% confidence interval. All five confidence intervals lie above zero, supporting positive aggregate gains at the nominal 5% level for each comparison.

BaselineMean score (%)AIM gain (pp)95% bootstrap CI for gain (pp)
ScientistOne50.87+3.49[0.79, 6.43]
AdaEvolve44.83+7.75[1.46, 13.50]
AIRA (MCTS)41.92+10.67[5.26, 16.44]
Arbor (max depth = 3)41.29+13.07[5.54, 20.48]
DeepScientist33.47+20.89[12.38, 27.93]

Regarding Data Select IFEval, its large improvement contributes substantially to the aggregate gain, but AIM outperforms ScientistOne on four of the five tasks. A particularly large gain on one task does not invalidate improvements on the others or, by itself, establish that the aggregate gain is statistically insignificant. The bootstrap analysis above directly quantifies uncertainty in that aggregate gain.


#5 Solution Auditor ablation study

Thank you for this suggestion! We separately evaluate the contributions of filtering and idea reconstruction on Flash Attention, using three independent runs with five branches and five iterations.

When we disable idea reconstruction, filtering remains active and the Solution Auditor skips reconstructing mismatched ideas from their implementations. When we disable filtering, the Solution Auditor still flags problematic results and reconstructs mismatched ideas, while flagged entries remain in the research pipeline.

The table reports the mean final best score for the selected solutions, with standard error across three runs. Removing idea reconstruction reduces the mean from 86.13% to 83.95%, a drop of 2.18 percentage points. Removing filtering reduces the mean to 85.25%, a drop of 0.88 percentage points.

ConfigurationFlash Attention score (%)
Full AIM90.5 ± 0.9
(--) idea reconstruction84.0 ± 1.1
(--) filtering85.3 ± 3.6

Dear Reviewer TwDv,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Reviewer_B2hqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper contrasts with solution-driven auto research and proposes an idea-driven framework, AIM, where the search objectives are ideas. The paper identifies three key challenge and provides solutions to each. The evaluation results with SOTAs show AIM's capabilities in searching better solutions more efficiently. A formal analysis further provides guidance on how to choose between solution and idea-driven frameworks.

Critical strengths
  1. Identifying ideation in contrast to validation&evaluation as an important search angle. Formal understanding between the two that provides useful insights of which strategy wins in what cases.
  2. Identifying computation budget as a critical bottleneck that requires optimization.
Critical weaknesses
  1. Could you please provide a more detailed breakdown of the wall-clock time of baselines and AIM into LLM time and execution time (=end-to-end wall-clock - LLM time)? This helps understand where the efficiency improvement comes from, and the contribution of idea management & selection components versus the resource selection component.
  2. Could you please test at least one task on AIM and two best baselines (one from solution-driven and one from idea-driven approaches)? This helps understand the robustness of the framework on LLM models.
Questions for authors

Same as weaknesses.

4 comments

We thank the reviewer for the constructive feedback and for recognizing the importance of distinguishing ideation from solution validation & evaluation, the value of relevant theoretical discussions, and the motivation of our approach.


#1 Breakdown of wall-clock time into LLM time and execution time

Thank you for the suggestion. We measure AIM’s coordination overhead relative to its total wall-clock time TT. The overhead L=L1+L2L=L1+L2 includes the time spent on idea management and selection L1L1 and on solution auditing and resource planning L2L2. The remaining time T−LT-L includes solver LLM calls, code execution, and other runtime overhead. We average the measurements across three runs per task.

TaskTotal Time T (min)LLM Call Time L = L1 + L2 (min)Idea Mgmt. L1 (min)Audit & Plan L2 (min)Execution Time T − L (min)L / T (%)
Flash Attention269.6521.4816.185.28248.178.0
Radix Sort267.3220.8817.673.22246.427.9
FFT Rust257.9320.1817.352.82237.757.9
AES128 CTR179.8718.2716.551.72161.5810.2
Z-order Range Scan198.6019.7017.981.72178.889.9
Data Select IFEval512.2315.1813.671.52497.033.0
Moving MNIST World Model937.6814.6712.622.03923.001.6
NTT Butterfly417.4715.8213.032.77401.653.8
Huffman Canonical Decode407.2816.0013.502.50391.273.9
ICP Correspondence Step385.3215.3812.532.83369.924.1

AIM's coordination pipeline accounts for about 6% of total wall-clock time on average, ranging from 1.6% to 10.2% across tasks. Most of the runtime goes to work outside coordination, including solver LLM calls and the execution and evaluation of generated solutions. Within the coordination pipeline, the Agentic Surrogate and Acquisition account for about 85% of the time, while the Solution Auditor and Resource Planner takes up the other 15%.


#2 Robustness against model choice

We interpret the reviewer's request as asking whether AIM's performance holds with LLMs other than Gemini-3.1-Pro-Preview. We will evaluate AIM, ScientistOne (our strongest idea-driven baseline), and AdaEvolve (our strongest solution-driven baseline) on Flash Attention and Z-order Range Scan tasks using Opus 5.

Opus 5Flash AttentionZ-order Range Scan
AdaEvolve (solution-driven)97.5 ± 2.547.2 ± 1.1
ScientistOne (idea-driven)86.6 ± 7.343.0 ± 0.9
AIM98.8 ± 1.248.4 ± 1.9

We also evaluated Flash Attention using GLM-5.2.

GLM-5.2Flash Attention
AdaEvolve (solution-driven)40.5 ± 16.2
ScientistOne (idea-driven)48.3 ± 3.4
AIM73.8 ± 2.3

In all cases, AIM showed robustness against model choices, consistently outperforming the strongest baselines.

Dear Reviewer B2hq,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Reviewer_B2hqOpenReview ↗

Response to the authors

Dear authors,

Thank you for the time and effort in adding these experiments. They resolve most of my questions.

Dear Reviewer B2hq,

We are glad that our response resolved most of your concerns!

We greatly appreciate your time and effort in reviewing our work. Please let us know if you have any further questions!

Best regards,

The Authors.

Reviewer_vDnoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Agentic Idea Manager (AIM), a framework for automated research that works idea-driven or with other words manages research ideas as the primary unit of search. It combines an agentic surrogate for organizing and ranking ideas, an agentic BO-inspired acquisition mechanism for exploration-exploitation decisions, a solution auditor for detechting idea-solution mismatches and invalid solutions, and a resource planner for adaptive budget allocation. Across multiple AutoLab tasks, AIM outperforms the strongest baseline by 1.6% on system optimization and 4.9% on model development and CUDA, while also reaching competitive solutions substantially faster. The authors further provide a theoretical intuition arguing that idea-level search is particularly beneficial when a task contains many semantically distinct directions but only a small fraction of them are competitive.

Critical strengths
  • clear and well-motivated problem formulation and idea: Automated research is a recently developing field and the distinction between solution-driven and idea-driven seems reasonable. Another interesting point is the two-level exploration-exploitation mechanism on cluster and idea level
  • strong empirical evaluation and results: Comparisons are done on multiple AutoLab problems and different research tasks and a clear outperformance of the baselines is noticeable. The ablation study provides further insights into the contribution of the single proposed components
  • theoretical intuition is generally useful: The dependence on semantic coverage provides an interesting conceptual explanation when idea-driven search is advantageous
Critical weaknesses
  • the novelty of the individual components is somewhat unclear for me. Your method consists of so many parts: semantic clustering, LLM-based ranking, exploration-exploitation decision, auditing, adaptive resource allocation, etc.. Maybe you can elaborate further on which parts you took over from related work (and which related work exactly) and what is really novel in your approach.
  • theoretical results do not align with practice: First of all, the semantic coverage proposition relies on the strong assumptions that, among other things, a meaningful mapping from executable solutions to discrete researcg directions exists, a faithful realization of ideas exists, and competitive directions are exchangeable, if I understood it correctly. In addition, it was unclear for me whether the theoretical results are applied in practice. For example in Proposition 2, you derive a sufficient coverage to identify a near-optimal research direction with high probability, but for me it is not clear, whether the exploration-exploitation decision mechanism takes care to achieve this amount of coverage through enough exploration in practice. Generally, it wasn't clear whether there is a certain amount of forced exploration to avoid getting stuck in a local optimum?
  • statistical evidence somewhat limited when you only average over 3 independent runs in your experiments
  • in section 4, when describing the AIM, it was a bit difficult to understand all parts of the search space. To me, it was not clear, from which domains the variables come and how they can be presented formally. A running example or more domain information might be helpful

Minor:

  • typo in line 112: "with the our proposed method"
  • the text parts in Figure 2(b) are very small and hard to read
Questions for authors
  • can you explain more about the exploration-exploitation decision? Are there any restrictions for the LLM to explore at least with a certain probability? Did you track how many times exploration and exploitation was chosen at both levels? I would expect an LLM without any restrictions to over-exploit promising regions to achieve 'somehow good' solutions as quickly as possible. On the other hand, even better regions are missed out completely then, because of the lack of exploration. Have you thought about using a more structured way for the exploration-exploitation decision, like bandit-based approaches?
  • the improvement of the wall-clock runtime is promising, but does that automatically translate into improved cost efficiency? Can you report total LLM-calls, input/output tokens and estimated inference costs for AIM in comparison to the baselines?
5 comments

We thank the reviewer for their constructive feedback and for recognizing the clear motivation, interesting methodological ideas, significant empirical findings, and valuable theoretical discussion in our work. We address the reviewer's key concerns below.


#1 Novelty and relationship to prior work

Thank you for requesting this clarification! AIM's central contribution is to make idea management itself an autonomous part of the research agent by giving it the control over how research ideas are organized, assessed, and selected as evidence accumulates. On the other hand, previous methods automate research within predefined idea-management structures, such as refinement trees or beam-style retention.

What AIM adds. To our knowledge, AIM is the first framework to jointly delegate the evolving organization of research ideas and exploration–exploitation decisions to an agent in the manner described below.

  • Agentic Surrogate: AIM introduces an agent that autonomously reorganizes the entire idea pool into semantic clusters using experimental evidence. Previous approaches organize ideas through predefined structures, such as refinement trees in Arbor or beam-style retention in ScientistOne.
  • Agentic Acquisition: As you have recognized as a strength of our approach, the two-level exploration-exploitation mechanism allows the agent to adapt to the evolving organization of ideas and accumulated evidence. Previous approaches select candidates through greedy retention or predefined exploration heuristics.
  • Solution Auditor: Prior work, including ScientistOne, has established the importance of auditing and evidence traceability. AIM's contribution is to integrate auditing directly into the idea-management feedback loop, excluding invalid results from subsequent iterations to prevent misleading evidence from propagating into later decisions.
  • Resource Planner: Prior works like ScientistOne or AdaEvolve often allocate resources using predefined scheduling rules. AIM delegates the allocation decision to the agent itself, autonomously balancing broader parallel exploration against additional rounds of sequential feedback.

Together, these components introduce specific mechanisms for autonomously organizing, selecting, and maintaining research ideas throughout the experimental process. Thus, each component adds something novel to the field of automated research.


#2 Practicality concerns in theoretical assumptions and exploration

We greatly appreciate the reviewer for thinking deeply about this important problem and raising these questions! In fact, we shared these concerns and have already discussed how the assumptions relate to practice in Appendix F.8. To reiterate a couple of points, (1) within each solver branch, solution-level refinement helps the agent implement its intended idea faithfully. Also, (2) the Solution Auditor identifies idea-solution mismatches and reconstructs ideas from the implemented solutions before attributing scores and extracting lessons. These mechanisms improve implementation fidelity and evidence alignment, bringing practice closer to the theoretical assumptions.

Regarding "forced exploration", Appendix E.4, Figure 15 provides evidence that the agent can adjust exploration autonomously. On Flash Attention, the share of branches involving exploration at either level decreases from 75% at iteration 2 to 33% at iteration 4. Conversely, on FFT Rust, this share increases from 50% at iteration 5 to 61% at iteration 6, with exploration at both levels rising from 19% to 42%. These results show that AIM can renew exploration as evidence accumulates, without a fixed exploration quota.


#3 Limited statistical evidence

We followed AutoLab's evaluation protocol, whose leaderboard also ranks methods by their average performance over 3 runs. To quantify uncertainty beyond reporting means, we additionally analyze aggregate performance across the five long-horizon Model Development & CUDA tasks using an exact task-stratified paired bootstrap.

For each baseline, we resample matched run pairs with replacement within each task, compute the task-level AIM–baseline differences, and average these differences equally across tasks. We enumerate the bootstrap resamples with their corresponding probabilities and report the central 95% confidence interval. All five intervals lie entirely above zero, corresponding to a p-value of p<0.05p<0.05 for each comparison under the two-sided percentile-bootstrap test. This supports positive aggregate gains at the nominal 5% level for each comparison.

BaselineMean score (%)AIM gain (pp)95% bootstrap CI for gain (pp)
ScientistOne50.87+3.49[0.79, 6.43]
AdaEvolve44.83+7.75[1.46, 13.50]
AIRA (MCTS)41.92+10.67[5.26, 16.44]
Arbor (max depth = 3)41.29+13.07[5.54, 20.48]
DeepScientist33.47+20.89[12.38, 27.93]

#4 Variable notation and domain clarification

Thank you for the suggestion. Section 3 defines the idea space X\mathcal{X} as structured natural-language descriptions of candidate approaches and the solution space Z\mathcal{Z} as executable implementations. We agree that Section 4 would benefit from more explicit domains for the variables built on these spaces. Thus, we have added a notation table in Appendix B.1 to clarify that Pt⊂XP_t\subset\mathcal{X} is the finite pool of discovered ideas, CtC_t groups these ideas into semantic clusters, RtR_t assigns integer ranks to clusters and unevaluated ideas, and DtD_t records audited idea–score pairs in X×R\mathcal X\times\mathbb R. The remaining state consists of the fixed task context TT and textual implementation lessons MtM_t. Dispatch selects unevaluated ideas Qt⊆PtQ_t\subseteq P_t, with branch actions in {Explore,Exploit\mathrm{Explore},\mathrm{Exploit}}2{}^2 and an integer branch budget BtB_t.

As a concrete example, consider a Flash Attention task whose fixed context TT includes task specifications and a brief survey of previous works in the relevant domain.

SymbolDomain or RoleExample or Description
PtP_tFinite pool of ideas in X\mathcal{X}The pool contains 10 ideas to enhance the Flash Attention code.
CtC_tSemantic clusters of ideas in PtP_tThe Agentic Surrogate groups the 10 ideas into three clusters: Advanced SIMD & AVX-512, Cache & Memory Bandwidth, and Fast Math & Exponentiation.
RtR_tInteger ranks for clusters and unevaluated ideasThe surrogate ranks Advanced SIMD & AVX-512 as first, Cache & Memory Bandwidth as second, and Fast Math & Exponentiation as third. Within each cluster, ideas are also ranked.
DtD_tAudited idea–score pairs in X×R\mathcal{X}\times\mathbb{R}An earlier trial recorded two pairs: (idea 1, 0.780.78), (idea 2, 0.650.65).
MtM_tImplementation lessons in textEarlier trials found that the 32×6432 \times 64 tile reduced memory traffic but left some threads idle.
BtB_tInteger branch budgetAt iteration tt, Bt=4B_t=4 parallel solver branches are assigned.
QtQ_tUnevaluated ideas selected for implementationBased on CtC_t, the Agentic Acquisition selects Bt=4B_t=4 ideas: Qt=Q_t = {branch 1: idea 3, branch 2: idea 4, branch 3: idea 5, branch 4: idea 6}

#5 Exploration-Exploitation Decision Analysis

Great question! As noted in #2, AIM leaves the exploration rate to the agent, with no minimum probability or quota. We have already tracked all four combinations of cluster- and idea-level actions across iterations on five tasks (Appendix E.4, Figure 15). On Flash Attention, cluster-level exploration fell from 58% of branches at iteration 2 to 22% at iteration 7, while idea-level exploration stayed near 42–44%. On FFT Rust, exploration at both levels rose from 19% at iteration 5 to 42% at iteration 6. These patterns show that the agent is capable of adjusting how much it explores over time and at each level. However, whether those choices reach the most promising regions remains an open question, and measuring coverage can be very challenging because the full set of research directions is unknown. Nonetheless, we think this is a very interesting direction for future research.

Prior MCTS-based methods often use a bandit rule such as UCT, which gives less-visited branches a larger exploration bonus. Visit counts capture how often a branch has been sampled, but they do not directly capture whether it tests a mechanism semantically distinct from those already explored. Applying a standard bandit to AIM also raises a separate challenge. As new ideas enter the pool and the agent reorganizes clusters, historical rewards and visit counts may no longer describe the same research direction. A bandit policy that carries evidence forward as clusters evolve is another promising direction for future work.


#6 Token cost efficiency

We have already provided the token cost analysis in Appendix D.4. Table 5 reports the total number of LLM calls and the input and output token counts for each method. Also, Figure 10 complements this by comparing the token cost trend of AIM and ScientistOne.


#7 Typo and figure enhancement

Thanks for your detailed review! We have fixed the typo and enhanced the font size of Figure 2 (b).

Dear Reviewer vDno,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Comments on the paper

1 comment

Dear Reviewers, AC, SAC, and PC,

We sincerely thank you for the time and effort devoted to reviewing our manuscript.

Automated research requires deciding which ideas deserve costly implementation and evaluation. As experiments accumulate, a research agent must organize an expanding pool of ideas, select promising directions, and ensure that experimental results accurately reflect the ideas being tested.

Our paper identifies idea management as a distinct design problem in automated research. We introduce AIM, which delegates idea organization, assessment, exploration–exploitation decisions, and budget allocation to an agent. Our evaluation spans ten AutoLab tasks, and our theoretical analysis explains when explicit search over research directions can be beneficial.

We appreciate the reviewers' recognition of the following strengths.

  1. Clear motivation and an important research problem.

    "clear and well-motivated problem formulation and idea" — Reviewer vDno

    "Identifying computation budget as a critical bottleneck that requires optimization." — Reviewer B2hq

  2. Useful conceptual and theoretical insights.

    "The dependence on semantic coverage provides an interesting conceptual explanation when idea-driven search is advantageous" — Reviewer vDno

    "Semantic Coverage provides an explicit definition of research-direction coverage, giving the discussion of search breadth a formal basis." — Reviewer fKsf

  3. Identification of a consequential failure mode.

    "The paper identifies how mismatches between proposed ideas and actual implementations can contaminate subsequent research decisions." — Reviewer TwDv

  4. Broad empirical evaluation and informative analyses.

    "The evaluation includes multiple search frameworks, three independent runs, component ablations, and a solver substitution experiment." — Reviewer TwDv

    "The average performance across both task groups exceeds that of ScientistOne, suggesting that the framework’s gains are not limited to a single task." — Reviewer fKsf


Summary of Manuscript Revisions

We revised the manuscript in response to the reviews, with changes marked in blue.

  • [B2hq] Model substitution experiments (Section 5.4). We added experiments testing if AIM works with different model families: Opus 5 and GLM-5.2.
  • [TwDv, fKsf] Ablations beyond Flash Attention (Section 5.2, Table 3). We added component ablations on NTT Butterfly. Removing any component lowers the mean best score, with the largest reduction of 3.4 percentage points from removing Agentic Acquisition.
  • [B2hq] Runtime breakdown (Appendix D.5, Table 7). We added coordination-time measurements across all ten tasks. Coordination averages approximately 6% of total runtime, with idea management and selection accounting for approximately 85% of coordination time.
  • [vDno] Formal notation (Appendix B.1). We added a notation table specifying the domains and meanings of the idea and solution spaces, search state, selection actions, and resource budgets.
  • [vDno] Evaluation protocol (Section 5). We clarified that the three-run evaluation follows the AutoLab leaderboard protocol and reports means and standard errors.
  • [vDno] Typo and Figure enhancement (Figure 2 (b)). We fixed the typo and enhanced readability of the figure by increasing the overall font size.

Overall Response

We appreciate the reviewers' comments and suggestions. Our responses clarify the scope of the search categorization, AIM's relationship to prior work, and how its implementation supports the theoretical assumptions. The additional ablations and runtime measurements provide further evidence about the contributions and overhead of its components. Together, these analyses have strengthened our study of how agents can autonomously manage research ideas under limited experimental budgets.

We hope this summary is helpful. Thank you again for your thoughtful feedback and for managing the review process.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.