- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the management and exploration of research ideas in automated research under limited experimental budgets. It proposes Agentic Idea Manager (AIM), a framework comprising four components: an Agentic Surrogate that dynamically organizes ideas into semantic clusters and ranks their promise; an Agentic Acquisition module that balances exploration and exploitation at both cluster and idea levels, dispatches candidates to parallel solvers, and generates new ideas from experimental feedback; a Solution Auditor that filters invalid results and corrects mismatches between ideas and their implementations; and a Resource Planner that adaptively allocates a fixed experimental budget across iterations. Across ten AutoLab tasks covering system optimization, model development, and CUDA programming, AIM improves average performance over the strongest baseline by 1.6 and 4.9 percentage points in the two task groups, respectively, and reaches baseline performance up to 3.1 times faster. The paper also provides a theoretical analysis explaining why broader semantic coverage can be valuable when competitive research directions are sparse.
Critical strengths
The paper identifies three challenges—organization, selection, and implementation consistency—giving the method’s motivation clear and concrete targets.
The average performance across both task groups exceeds that of ScientistOne, suggesting that the framework’s gains are not limited to a single task. The main text also acknowledges AdaEvolve’s advantage on the data-selection task.
Semantic Coverage provides an explicit definition of research-direction coverage, giving the discussion of search breadth a formal basis.
Critical weaknesses
The paper categorizes search methods as either idea-driven or solution-driven, but this distinction lacks strong justification. It is therefore unclear whether the taxonomy comprehensively covers all search methods.
Both the main component ablations and the more fine-grained surrogate ablations focus on Flash Attention, making it difficult to determine whether the same mechanisms explain the gains on model-development and CUDA tasks.
Table 5 shows that AIM uses substantially more input tokens than some baselines. While the common-goal comparison in Fig. 10 is valuable, the paper does not compare all methods under a unified total token budget.
Questions for authors
See Weakness
3 comments
Dear Reviewer fKsf,
Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!
Best regards,
The Authors
Dear Reviewer fKsf,
Thank you again for your time and thoughtful feedback!
We understand this is a busy period, but with our discussion deadline coming to a close, we wanted to gently follow up. We have done our best to address your questions and would be happy to use the last minute of our discussion period to provide any further clarification. Could you please let us know if our responses addressed your concerns?
Thanks again, and we look forward to your response!
Best regards,
The Authors.
We thank the reviewer for the constructive feedback and for recognizing the clarity of our motivation, the significance of our empirical findings, and the value of our theoretical discussion. Below, we address the reviewer's key concerns.
#1Justification of the idea-driven vs. solution-driven categorizationThank you for this important question! First of all, we would like to highlight other reviewers' comments about the categorization
Moreover, Peirce's account of scientific inquiry provides conceptual support for this distinction. He separates forming an explanatory hypothesis (retroduction), deriving its testable consequences (deduction), and assessing those consequences against experience (induction) [1]. In the automated research settings we study, the problem is already specified, and ideas are realized through executable implementations. Experimental feedback can therefore guide search at either level. Idea-driven search uses results to select ideas for subsequent implementation, while solution-driven search uses results to select and refine executable candidates.
Our categorization captures which of these two spaces serves as the primary search space. A method can explicitly search over ideas and delegate implementation to a solver or search directly over executable solutions. These possibilities encompass search over the idea and solution spaces defined in our formulation, including hybrid approaches. Therefore, Peirce's account supports the conceptual separation between these levels, while our categorization makes that separation operational for comparing automated research methods.
[1] Peirce, C. S. (1908). “A Neglected Argument for the Reality of God.”
#2Ablations on Model-development & CUDA tasksGreat suggestion! We extend the ablation study to NTT Butterfly, a longer-horizon Model Development & CUDA task. We report the mean best score (%) and standard error across 3 independent runs. The table below shows the ablation study results on both Flash Attention and NTT Butterfly.
#3Token cost and performanceAIM's input-token usage reflects its use of accumulated research context, including candidate ideas, evaluation history, and implementation lessons. This context allows the agent to compare research directions, revisit promising ideas, and use previous outcomes to guide subsequent experiments. The additional cost thus supports the agent's decisions about where and how to spend the experimental budget.
To make this trade-off more explicit, we will report AIM's score and cumulative token usage after the first iteration alongside the baselines' final results. The following comparison uses Flash Attention, with final-run values from Table 5.
After its first iteration, AIM achieves 83.6%, exceeding the final scores of Arbor (77.2%), AIRA-MCTS (77.9%), and AIRA-EVO (76.3%), at an estimated cost of 43.19 USD, comparable to their 27.40 - 39.84 USD. This comparison clarifies how much of AIM's advantage emerges at a comparable monetary cost. Its subsequent iterations spend additional tokens to refine the search and reach a final score of 90.5%.
However, we certainly agree that reducing this token overhead is an important direction for future work. In particular, instead of keeping all the accumulated research context from repeated rounds, we may investigate ways to compress experimental histories into concise evidence summaries, retrieving only the context relevant to each decision, and incrementally updating cluster assessments instead of repeatedly processing the entire idea pool. These changes could reduce input-token usage while retaining the accumulated evidence that informs AIM's research decisions.