acceptodds
Under review as a conference paper at ICLR 2027

Structure and Dynamics of Agentic Innovation

Abstract

We study scientific innovation as a dynamical system: the state is a citation network of research ideas distilled from published papers, and the dynamics are LLM agents that navigate the network and write new ideas (and reference links) back into it. Agents propose ideas freely; each new idea is then scored by whether a similar paper appears in a later conference, published after both the papers in the network and the agent model's training cutoff. We create multiple agents to navigate a 16,208-node network built from NeurIPS, ICLR, ICML, and AAAI papers (2020–2024) and change the number of corpus papers an agent is allowed to read, from 100 up to the whole corpus. We find three things. First, more reading is not better: the share of ideas that later papers go on to realize rises and then falls, peaking at 800 papers, while both extremes sink to the same floor, agents given no papers doing no better than agents free to read all 16,208. The number of ideas produced follows the same curve. Second, team size buys only throughput: the hit rate is flat from 10 to 150 agents while the total scales linearly, because the agents hardly interact—in 10,300 actions, one agent cited another's idea exactly once. Third, the human citation graph is not what helps them: deleting all 80,962 edges makes generalists better, while removing semantic search makes them worse. Agents move through the network by meaning rather than by links, and the links they write back ignore a paper's popularity and favor recent frontier work. Reading scope—not team size, not citation structure—is the lever on the quality of machine-generated research ideas.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent systems, research ideation, scientific discovery, agent memory, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5PFBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This study aims to evaluate the idea generation piece of AI Scientist frameworks. This is not a new AI Scientist framework, but focuses on isolated study of the idea generation mechanism, trying to answer questions like: if given more papers does it correlate to better ideas? Does idea generation improves as number of agents are scaled? Is citation network more advantageous or semantic search? The agents here do not evaluate an idea by running experiments. They only propose new ideas. A new idea is marked valid if a subsequent research paper is found, which published after the training data cutoff of the LLM used. The authors find that focused literature access produces better ideas than unrestricted access, increasing the future-paper matching rate from 14% to 31%. Increasing the number of agents improves idea quantity but not quality, largely because agents rarely collaborate. Semantic search also outperforms citation based exploration.

Critical strengths
  1. The framework and experiment design is very elegant in terms of isolating the research question.
  2. The experiments show the importance of citation network and number of papers.
  3. Empirical findings are insightful.
Critical weaknesses
  1. Presence of a research paper does not guarantee the idea is scientifically sound. It could just mean that it is an obvious research direction or a low hanging fruit based on a previous study. Quality of the generated idea is difficult to evaluate this way.
  2. The multi-agent collaboration conclusion seems a little weak. Given the construction of the framework, it is rare that agents will find other agents work. It is not a clear indicator of the limitation of multi agent collaboration rather than being a limitation of their shared memory mechanism.
  3. The sweet spot of number of papers can be a little misleading considering the evaluation was done using only one model. The argument that N number of paper is needed for context and more than that can cause noise, could be very model specific artifact. There are no experiments to show otherwise.
  4. The conclusion on constraints beats freedom is weak since the success or failure is evaluated if a research paper is published subsequently. This ties to the weakness 1, where constraints can lead to next obvious idea from a research paper and the scoring is biased towards that.
  5. The metric used here cannot distinguish between ideas that are genuinely flawed or unviable and sound ideas that simply haven't been pursued by human researchers yet.
Questions for authors
  1. Can the authors provide some experiments to show, these conclusions hold across models?
  2. Out of the ideas generated by the agent, which according to the current metrics are evaluated as failure, is there a way to distinguish ideas actually worth pursuing and idea not worth it? Maybe a controlled experiment which goes beyond idea generation into execution can provide some insights on this.
  3. Can the authors show some statistical replication?
  4. Is there possibility of contamination since there can be version of the same work in different avenues, published before the cut off?
  5. Given that removing citation edges actually improves performance, does this suggest that explicitly modeling papers as a relational graph offers no significant advantage for the scientific ideation task?
Reviewer_hwVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how LLM agents generate research ideas using a shared citation network built from 16,208 AI conference papers. The empirical study investigates how much literature agents can access, how many agents participate, and how they find papers. Generated ideas are then compared with later publications using literature search and an LLM-based judge. The results suggest that agents produce ideas that match later research more often when their reading is limited to a set of papers related to a specific topic, compared with agents allowed to explore the entire collection. Adding more agents mainly increases the total number of ideas, without consistently improving the matching rate. Agents rarely cite ideas generated by their teammates, suggesting that they mostly work independently rather than develop ideas together.

Critical strengths

The authors builds a large citation network from 16,208 papers and explores several experimental dimensions, including reading scope, team size, and navigation tools, providing insights into how these choices affect research idea generation.

Critical weaknesses
  1. This work misses important related work, such as VIRSCI [1] and ResearchAgent [2], and provides no comparison with established systems for research idea generation. Such comparisons would help clarify its contribution and whether its findings also hold for systems with explicit collaboration.
  2. The motivation behind the pipeline design is unclear. Agents use a predefined set of actions, such as search and browse, under strict reading and memory constraints. The paper does not adequately explain why these choices can provide a convincing model of how researchers explore literature and develop ideas. The design for multiple agents is also underdeveloped: agents use the same action interface and interact only through the shared graph, without a dedicated process for discussing, evaluating, or refining ideas together. It is therefore unclear how meaningful scientific collaboration is expected to emerge from this setup.

[1] Su, Haoyang, et al. "Many heads are better than one: Improved scientific idea generation by a llm-based multi-agent system." Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers). 2025.

[2] Baek, Jinheon, et al. "Researchagent: Iterative research idea generation over scientific literature with large language models." Proceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: human language technologies (volume 1: long papers). 2025.

Questions for authors
  1. What motivated the choice of retaining only the latest 20 action and result pairs in each agent’s memory? Is there evidence that the advantage of restricting access to literature is robust to this choice, or should the finding be interpreted as specific to this memory budget?
  2. Could the authors clarify what CCF-A and CCF-B refer to in Table 1?
  3. How did the authors verify that the generated summaries accurately capture each paper’s main contribution, given that these summaries determine semantic search and agents’ reading scopes?
Reviewer_1jwNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models scientific innovation as a dynamical system: the state is an idea citation network distilled from 16,208 top AI conference papers (2020-2024.8), and the dynamics are gpt-5 agents exploring and writing back to new nodes on the network through searching, browsing, jumping, adding or deleting links, and generating new ideas. Only when a paper published after June 2025 has implemented a certain generated idea can it be considered a "successful prediction", and it will be scored by the LLM judges. The experiment found that the number of papers that the agent can read shows an inverted U-shaped effect. The hit rate is the highest when there are approximately 800 papers, while the performance of not reading papers is comparable to that of reading the entire database. The increase in team size from 10 to 150 only boosts output without improving the quality of individual ideas, as agents hardly quote each other's ideas. Deleting all reference edges actually improves performance, while disabling semantic search leads to a decrease in accuracy, indicating that the agent navigates by semantics rather than reference links. Based on this, the author believes that the reading range is a key lever affecting the quality of ideas generated by machines. However, each condition was only run once and the sample size was small, so the statistical evidence for some conclusions is limited.

Critical strengths

A controllable testbed with a leakage-guarded, outcome-based evaluation.

The paper turns a researcher’s “information diet” into a directly manipulable variable (read scope m, team size N, channel ablations), an intervention that is impossible with human scientists. Instead of relying on human or LLM preference judgments of the ideas themselves, it scores ideas by whether papers published after the model’s knowledge boundary (≥ 2025-06-01, chosen via memory probes) realize them, excludes corpus matches and near-duplicates, and anchors all comparisons to a zero-reading floor (C0, 13%). This is consequential because credible, hard-to-game evaluation is the main bottleneck in automated research ideation, and the protocol is a reasonable template that others could build on.

Critical weaknesses

1. Insufficient Evidence to Support the Main Claims. As an analysis paper without methodological contributions, its value depends on the reliability of its empirical findings. However, each condition uses only a single seed, with 50–83 ideas and 7–23 hits, while the reported run-to-run variance (±7 percentage points) is comparable to most observed differences. Key claims, including the inverted-U effect of reading scope, the benefit of removing citation edges, and the importance of search, lack statistical support. No confidence intervals or cluster-aware significance tests are provided. Thus, the strong conclusions in the title and abstract are not adequately justified.

2. Immature Evaluation Protocol and Limited Mechanistic Insights. The evaluation relies on an uncalibrated GPT-5-mini judge, limited retrieval coverage, and small probe sets. Important baselines, such as single-shot generation, trend extrapolation, and random reading scopes, are missing. Moreover, the findings remain largely descriptive: the team-size effect follows from the system design, the inverted-U explanation is speculative, and the citation analysis lacks temporal controls. Without a validated benchmark, convincing mechanistic evidence, or a method derived from the findings, the paper's contribution to ICLR remains limited.

Questions for authors
  1. Can you provide results over multiple seeds and topic draws (with cluster-aware confidence intervals, e.g., bootstrap over agent threads) for the key comparisons: m=800 vs. 400/1600, C0 vs. C1, and full graph vs. no-edges?
  2. Do the main rankings hold with a different judge model (or a non-OpenAI judge) to rule out same-family bias?
  3. Does a random subset of 800 papers (not nearest to the topic anchor) perform differently from the nearest-800 region? Without this, the effect cannot be attributed to “situated” reading rather than simply a smaller search space.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.