acceptodds
Under review as a conference paper at ICLR 2027

IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation

Abstract

Large Language Models (LLMs) have significantly advanced the automation of scientific discovery. However, existing systems share a core limitation: they optimize ideas independently for either Quality or Diversity. This often yields closely related ideas or large sets of trivial, unsound, or unclear concepts. We instead argue that research ideation should be framed as a conjunction of Quality and Diversity objectives, i.e., a Quality-Diversity (QD) search. Accordingly, we introduce IDEAgent, a multi-agent framework that tracks idea evolution through lineages. We jointly drive Quality using multi-objective feedback for repair and refinement, while Diversity is achieved through lightweight sequential memory and explicit comparison against completed ideas, historical variants, and rejected proposals. To evaluate this joint objective, we develop Yield, a metric that computes the largest mutually diverse set of ideas satisfying predefined quality thresholds. Across 32 topics spanning 8 CS domains, IDEAgent outperforms the best baseline by on Yield, while achieving Yield of at least on as many topics. We further analyze quality improvements, showing that repair and refinement improve logical rigor and clarity while preserving non-obviousness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for Scientific Discovery, Multi-Agent Systems, Quality-Diversity Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1YCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IDEAGENT, a multi-agent framework that tracks idea evolution through lineages. They jointly drive Quality using multi-objective feedback for repair and refinement, while Diversity is achieved through lightweight sequential memory and explicit comparison against completed ideas, historical variants, and rejected proposals. To evaluate this joint objective, this paper also develops Yield, a metric that computes the largest mutually diverse set of ideas satisfying predefined quality thresholds.

Critical strengths

It introduces IDEAGENT, a multi-agent framework that tracks idea evolution through lineages using multi-objective feedback for repair and refinement, while Diversity is achieved through lightweight sequential memory and explicit comparison against completed ideas

Critical weaknesses

The evaluation relies almost entirely on LLM judges, without human expert validation. The agreement between the two external judges is also relatively low, especially for soundness (κ = 0.268), which is a key component of Yield. This raises concerns about the reliability of the reported improvements.

The proposed framework mainly combines existing techniques, including multi-agent evaluation, sequential memory, and iterative refinement. While the Quality-Diversity formulation is interesting, the technical contributions beyond existing research idea generation methods are relatively limited.

Questions for authors

Most baselines are simplified generation strategies rather than existing research idea generation systems. Although the authors provide additional scaling experiments, comparisons with stronger methods under matched inference costs are still missing.

The paper does not evaluate whether the generated ideas are actually novel with respect to existing literature or whether they can be successfully implemented. As a result, improvements in the proposed Yield metric may not translate into better research ideas in practice.

Reviewer_UYxQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper frames LLM-based research ideation as a Quality-Diversity (QD) search. Under a fixed budget, the goal is to produce as many ideas as possible that are each high quality (non-obvious, sound, and clear) and mutually distinct. The proposed IDEAGENT is a multi-agent framework that generates ideas sequentially and tracks each idea’s evolution as a lineage. Internal judges score each draft on quality and compare it against archives of active, historical, and rejected ideas. Ideas that pass a quality gate are accepted. Near-misses are repaired, and accepted ideas can be further refined, both guided by targeted feedback.

Critical strengths

The idea generation approach based on quality and diversity is intuitive and well-motivated.

Critical weaknesses
  1. Presentation and clarity

No overview figure. The method involves multiple agents, memory stores, processing stages, and routing mechanisms, yet these components and their interactions are described only in text. Combined with the numerous evaluation metrics, this makes the paper extremely difficult to follow.

Inconsistencies. The text does not match Figure 1 (eg., 3.2/+1.9 vs. 3.3/2.0).
There are several typos and grammatical errors in the main text (e.g., 70: “designed to measures”, 377L: “candiates”, 172L: "an explicit instruction to pivot to abandon that failing mechanism")

  1. Reliability of the experiments
  • Circular evaluation. IDEAGENT filters, repairs, and refines ideas against the same quality dimensions the final evaluation measures, raising suspicion that the gain is from evaluation alignment rather than actual diversity.

  • Unjustified hyperparameters. Many constants (τ=60, δ=20, refinement targets 90/80/80, weights 0.7/0.2/0.1, K_aux=2, M=5) are neither motivated nor ablated. eg., refinement targets 90/80/80, weights 0.7/0.2/0.1, K_aux=2, M=5 are neither motivated nor ablated.

  • Inadequate baselines. All baselines are the authors' own variants. Comparisons with established research idea generation systems are needed to assess the method’s contribution relative to prior work, such as VirSci [1], ResearchAgent [2], and Chain of Ideas [3] ...

  • Yield itself is never validated. The paper provides no experiment showing that Yield correlates with human researchers’ judgments or practical usage. The human study in Appendix B assesses only three single-dimension quality scores and evaluates neither diversity nor Yield. Moreover, the internal judges filter and optimize along the same dimensions that Yield measures. As a result, it cannot be determined whether a high Yield indicates genuinely good ideas or merely ideas aligned with the scoring criteria

  • Self-built, narrow evaluation. All 32 topics are self-constructed, with no quality validation, no temporal cutoff analysis for possible contamination, and no practical examination of these ideas. Extra evaluation on public benchmarks (e.g., IdeaBench[4], LiveIdeaBench[5]) or comprehensive examination of the self-curated dataset is needed.

[1] Su, Haoyang, et al. "Many heads are better than one: Improved scientific idea generation by a llm-based multi-agent system." Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers). 2025.

[2] Baek, Jinheon, et al. "Researchagent: Iterative research idea generation over scientific literature with large language models." Proceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: human language technologies (volume 1: long papers). 2025.

[3] Li, Long, et al. "Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents." EMNLP (Findings). 2025.

[4] Guo, Sikun, et al. "Ideabench: Benchmarking large language models for research idea generation." Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2025.

[5] Ruan, Kai, et al. "Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context." Nature communications 17.1 (2026): 3625.

Questions for authors

see weakness

Reviewer_dYKCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IDEAGENT, a multi-agent framework that formulates automated scientific research ideation as a Quality-Diversity (QD) search problem. Recognizing that current LLM-based ideation methods optimize for quality or diversity in isolation—leading to conceptual collapse or minor variations—IDEAGENT drives quality through multi-objective feedback (repair and refinement) while maintaining diversity via sequential memory and explicit comparisons against completed, historical, and rejected proposal archives. To evaluate joint QD performance, the authors introduce Yield, a metric that calculates the maximum size of a mutually diverse subset of generated ideas meeting strict quality thresholds (non-obviousness, soundness, and clarity). Across 32 Computer Science topics, IDEAGENT demonstrates significant improvements in Yield over baseline approaches.

Critical strengths
  1. Strong Conceptual Reformulation: Reframing scientific ideation as a Quality-Diversity (QD) search addresses a critical pain point in AI-driven discovery—the phenomenon of "conceptual collapse" where LLMs repeatedly produce superficial variations.

  2. Well-Designed Evaluation Metric (Yield): The introduced Yield metric elegantly prevents "metric gaming" by taking the maximum clique of ideas that simultaneously pass hard quality gates and satisfy pairwise diversity constraints.

  3. Pragmatic Memory and Archive Architecture: Utilizing compact semantic signatures/summaries (Problem, Mechanism, Value, Assumptions, Effect) for memory stores (Active, Historical, Rejected) effectively prevents context bloat during sequential ideation.

  4. Thorough Downstream and Cost Analysis: The paper provides comprehensive empirical validation across 32 CS topics, including cost-per-successful-topic (CST) analysis, baseline scaling experiments, and a human preference check.

Critical weaknesses
  1. Both agent feedback loops and final jury evaluations rely heavily on closed-source LLMs (GPT-5.6, Gemini 3.1, Claude 5). Despite inter-annotator checks, LLM-as-a-judge remains inherently susceptible to self-preference, style bias, and variance when evaluating complex academic novelty or logical soundness.

  2. From an agentic system perspective, IDEAGENT’s building blocks—multi-agent debate, memory archives, structured summaries, and repair loops—are well-established paradigms. The contribution lies primarily in assembling them into a QD-inspired pipeline rather than introducing a novel algorithmic primitive.

  3. Evaluation is conducted strictly at the text/proposal level. Without code execution, empirical experiments, or theoretical proofs, it remains unverified whether ideas rated high in "soundness" translate to truly workable scientific breakthroughs in practice.

Questions for authors
  1. As noted in Appendix A.4, smaller open-source models struggled with initial soundness. How does IDEAGENT perform if open-source models (e.g., Llama-3/Qwen-3 variants) are used as Ideators while leaving the Judge/Critic to proprietary LLMs?

  2. The experiments are restricted to 8 CS domains. How well do the prompt-based evaluation rubrics (e.g., non-obviousness or soundness) transfer to non-CS fields like biology or physics, where domain knowledge grounding is significantly different?

  3. How sensitive is the Yield metric to variations in the quality thresholds (τN,τS,τC=60\tau_N, \tau_S, \tau_C = 60)? Does altering these thresholds change the relative ranking among baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.