- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This study aims to evaluate the idea generation piece of AI Scientist frameworks. This is not a new AI Scientist framework, but focuses on isolated study of the idea generation mechanism, trying to answer questions like: if given more papers does it correlate to better ideas? Does idea generation improves as number of agents are scaled? Is citation network more advantageous or semantic search? The agents here do not evaluate an idea by running experiments. They only propose new ideas. A new idea is marked valid if a subsequent research paper is found, which published after the training data cutoff of the LLM used. The authors find that focused literature access produces better ideas than unrestricted access, increasing the future-paper matching rate from 14% to 31%. Increasing the number of agents improves idea quantity but not quality, largely because agents rarely collaborate. Semantic search also outperforms citation based exploration.
Critical strengths
- The framework and experiment design is very elegant in terms of isolating the research question.
- The experiments show the importance of citation network and number of papers.
- Empirical findings are insightful.
Critical weaknesses
- Presence of a research paper does not guarantee the idea is scientifically sound. It could just mean that it is an obvious research direction or a low hanging fruit based on a previous study. Quality of the generated idea is difficult to evaluate this way.
- The multi-agent collaboration conclusion seems a little weak. Given the construction of the framework, it is rare that agents will find other agents work. It is not a clear indicator of the limitation of multi agent collaboration rather than being a limitation of their shared memory mechanism.
- The sweet spot of number of papers can be a little misleading considering the evaluation was done using only one model. The argument that N number of paper is needed for context and more than that can cause noise, could be very model specific artifact. There are no experiments to show otherwise.
- The conclusion on constraints beats freedom is weak since the success or failure is evaluated if a research paper is published subsequently. This ties to the weakness 1, where constraints can lead to next obvious idea from a research paper and the scoring is biased towards that.
- The metric used here cannot distinguish between ideas that are genuinely flawed or unviable and sound ideas that simply haven't been pursued by human researchers yet.
Questions for authors
- Can the authors provide some experiments to show, these conclusions hold across models?
- Out of the ideas generated by the agent, which according to the current metrics are evaluated as failure, is there a way to distinguish ideas actually worth pursuing and idea not worth it? Maybe a controlled experiment which goes beyond idea generation into execution can provide some insights on this.
- Can the authors show some statistical replication?
- Is there possibility of contamination since there can be version of the same work in different avenues, published before the cut off?
- Given that removing citation edges actually improves performance, does this suggest that explicitly modeling papers as a relational graph offers no significant advantage for the scientific ideation task?