acceptodds
Under review as a conference paper at ICLR 2027

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

18 views

Abstract

Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%. Code and data will be released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Research Ideation, Multiagent Systems, Reinforcement Learning, Automated Scientific Research, Cross Domain Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_62sQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats research ideation as a standalone task: given a problem definition and a challenge, produce a research proposal. IdeaScientist uses an orchestrator and three sub-agents. A gap finder identifies a shared limitation in related work, an innovator looks for mechanisms from other fields that could transfer, and a report writer turns the result into a full proposal. Each role is trained separately with LoRA and asynchronous GRPO, using a reward that combines a reference-free LLM score, an LLM comparison against a reference built from the human paper, and citation F1. The authors also build the Svalbard Idea Vault, 2.77M ideas decomposed from 4.7M arXiv papers. They use papers after a January 2026 cutoff as training and test targets, and retrieval at evaluation time is restricted to pre-cutoff literature. On Qwen3.6-27B, the trained system beats the best open autoresearch baseline by 14% and also scores above Claude Code (Claude-4.8-Opus) and Codex (GPT-5.4), mainly because of higher novelty scores.

Critical strengths
  1. Studying ideation on its own makes sense. With end-to-end systems it really is hard to tell whether gains come from the idea or from the execution.
  2. Training each role on its own reference artifact is a reasonable way around the sparse reward of the full pipeline.
  3. The paper includes some analysis of judge reliability (cross-judge agreement, self-preference) and a human pairwise study.
Critical weaknesses
  1. The training reward conflicts with the novelty goal. r_ref rewards agreement with a Qwen-written reconstruction of the target paper, so the agents are trained to reproduce the human paper’s solution. This sits uneasily with the paper’s main claim that IdeaScientist’s advantage comes from novelty. It would help to explain how the reward avoids penalizing valid but different solutions, and to check whether r_ref agrees with human judgments of quality.
  2. The same judge is used for training and evaluation. Qwen3.6-27B builds the references, gives the GRPO reward, and is the main judge in Table 1, so part of the gain is expected by construction.
  3. The risk of contamination and leakage is underestimated. The targets come from January–June 2026, while Qwen3.6, Claude-4.8, and GPT-5.4 were released between March and May 2026. A title-only probe on ten papers is weak. Challenges rewritten from the full paper may already point toward its solution, and the random split within one six-month window likely puts closely related papers in both train and test.
  4. The ablations are limited. The sub-agent ablation reports only novelty. The retrieval ablation’s main gain is on the In-domain metric, which is biased toward cross-domain transfer to begin with. There’s no RL vs. SFT comparison.
Questions for authors
  1. Without the In-domain column, does Overall still favor IdeaScientist over Claude Code and Codex?
  2. With Claude or GPT as the main judge, does Trained still beat Base?
  3. How well do the training reward and its components agree with human judgments? Is there an ablation over the components and weights?
10 comments

We thank the reviewer for recognizing the motivation for studying ideation on its own, the stage-matched training signal for sparse rewards, and the judge-reliability analysis! We address each concern below.

W1. The reference-grounded reward may penalize valid but different solutions and thus conflict with the novelty goal.

Thank you for the question! The report-writer reward consists of four components: a reference-free quality score that judges the proposal on its own, a reference-grounded score that compares the proposal with the human-written solution, a citation score, and a process term penalizing protocol violation. The reference-grounded component therefore contributes only 28% of the training reward.

The reference-grounded component is designed to reward substantive research decisions: every target paper in our data is a methodology paper that proposes a new method, so the reference-grounded component rewards the kind of substantive methodological decisions that such papers make, rather than surface similarity. We agree that such a reward could in principle penalize a different but equally novel method. The reward definition avoids this with the use of reference-free component rewarding the standalone quality of every proposal independently of the reference.

Empirically, we find no negative association between reference agreement and novelty, here measured as the judge's novelty against all literature, and for human annotations as human-rated novelty:

Proposals comparedCorrelation between reference agreement and novelty
IdeaScientist-Base+0.04
IdeaScientist-Trained+0.07

All correlations are close to zero and none is significantly negative, so proposals that agree more with the human reference are not less novel, and training that raised reference agreement did not lower novelty on the same papers.

Finally, we pointed out that similarity to the realized paper is also the main training or evaluation signal in recent ideation work such as GIANTS and DeepInnovator, as noted by Reviewer 3ctU and in Zonglin Yang’s public comment. We improved by defining reference-free evaluation rubrics.

W2 / Q2. Using the same judge for training and evaluation may inflate the gains; does Trained still outperform Base under Claude or GPT judges?

Thank you for this question. We would like to clarify that the improvement from training is not tied to the training judge. The cross-judge evaluation in Section 5.3 (Figure 4) scored every IdeaScientist proposal with Claude-4.8-Opus and GPT-5.4, which are never used anywhere in training, as well as Gemma-4-31B and GPT-OSS-120B. Since the paper reported only the agreement between judges, we additionally report here the training effect under each judge. Agreement with the research direction of the held-out human paper, the quantity that RL optimizes, improves under every judge:

JudgeIdeaScientist-BaseIdeaScientist-Trained
Qwen3.6-27B (training judge)0.2440.299
Claude-4.8-Opus0.4800.557
GPT-5.40.3370.409
Gemma-4-31B0.2810.388
GPT-OSS-120B0.2490.304

Under both frontier judges, Trained also improves average proposal quality over Base (68.8 vs. 70.0 under Claude-4.8-Opus and 74.5 vs. 76.0 under GPT-5.4). In addition, the harness comparisons in Table 1 also hold under the frontier judges: IdeaScientist-Base exceeds every one of the seven open baselines in novelty by +12.7 to +21.8 points under Claude-4.8-Opus and by +5.1 to +15.0 points under GPT-5.4.

W3. The evaluated models may have memorized post-cutoff papers, and training and test targets may leak into each other.

We consider January 1, 2026 a reasonable cutoff that balances model capability with contamination risk for three reasons. First, GPT-OSS-120B was released on 2025-08-05, before our 2026-01-01 cutoff, so it cannot have seen any post-cutoff target paper during training. Second, as reported in Appendix E.2, we manually queried our evaluated models about sampled post-cutoff target papers, providing only the paper title without retrieval, and none of the models identified or described any of them as known work.

For leakage between training and test targets, we manually reviewed all test topics and deduplicated the training and test targets using both LLM-based and manual checks to ensure that no test paper overlaps with a training paper.

W4. The ablations are limited: the sub-agent ablation reports only novelty, the retrieval ablation's gain is mainly In-domain, and there is no RL vs. SFT comparison.

Thank you for the question! We extend our ablation studies to the other judge dimensions. The table below reports all six quality dimensions, their average, and novelty against all literature for the test papers, using the two backbones in the ablation without training:

BackboneConfigurationRelevanceClaritySpecificityActionabilitySoundnessImpactAverage qualityNovelty against all literature
Qwen3.6-27BReport writer only95.585.481.975.659.270.778.059.5
Gap finder and report writer97.290.186.478.764.275.382.063.4
Full pipeline (with innovator)96.289.980.677.362.873.480.066.0
Gemma-4-31BReport writer only95.184.271.970.257.867.174.449.6
Gap finder and report writer94.588.073.572.260.170.676.555.8
Full pipeline (with innovator)94.989.174.573.360.370.677.161.4

We can see that the gap finder and report writer together contribute mostly to the other dimensions, while the innovator provides a strong boost in novelty. The other judges are not affected much by whether the innovator is included, because they care mostly about the quality of the writing and presentation without judging the novelty of the idea itself.

We also ran the requested RL vs. SFT comparison for all three producer roles, fine-tuning Qwen3.6-27B on the best of eight base-policy trajectories per group (rejection-sampling fine-tuning, RFT) and evaluating all policies on the same held-out validation instances with the same tools and judge. We found that GRPO outperforms RFT on every role by a large margin:

RoleBaseRFTGRPO (ours)
Gap finder52.757.566.4
Innovator51.454.068.2
Report writer77.981.188.2

Q1. Without the In-domain column, does Overall still favor IdeaScientist over Claude Code and Codex?

Without the In-domain column, IdeaScientist remains the best system on every open backbone (74.1 vs. 67.8 on Qwen3.6-27B, 69.5 vs. 64.2 on Gemma-4-31B, and 67.8 vs. 64.3 on GPT-OSS-120B). On matched frontier backbones, it outperforms Claude Code on Claude-4.8 (81.5 vs. 77.4), while Codex scores higher on GPT-5.4 (78.3 vs. 75.2). We recompute Overall from the submitted Table 1 as the mean of proposal-quality Avg and the mean of All and Cutoff novelty. IdeaScientist uses Trained on Qwen3.6-27B and Base on the other backbones:

BackboneIdeaScientistBest other system
Qwen3.6-27B74.167.8 (DeepScientist)
Gemma-4-31B69.564.2 (AI-Researcher)
GPT-OSS-120B67.864.3 (AI-Researcher)
Claude-4.881.577.4 (Claude Code)
GPT-5.475.278.3 (Codex)

Q3. How well do the training reward and its components agree with human judgments, and were the components and weights ablated?

We found that the reward agrees with human judgments, mainly through its reference-free component. We matched human-annotated proposals with valid stored report-writer rewards and computed proposal-level Pearson correlations with the two annotators’ mean scores, using paper-clustered 95% confidence intervals:

Reward signalCorrelation with human quality (six dimensions)Correlation with human novelty
Reference-free quality0.52 [0.44, 0.60]0.36 [0.27, 0.44]
Reference agreement0.07 [−0.06, 0.19]−0.01 [−0.15, 0.12]
Citation F10.14 [0.03, 0.24]0.19 [0.08, 0.28]
Full report-writer reward0.33 [0.21, 0.43]0.24 [0.12, 0.35]

The components play complementary roles. The reference-free component carries the agreement with human judgments of quality (0.52) and novelty (0.36), while reference agreement measures a different property, agreement with the research direction that the field actually took, and is neither associated with human quality nor with human novelty, so it does not penalize novel proposals. Its effect is instead visible in what training improves: agreement with the held-out human solution rises significantly under all five judges, including two frontier judges (Claude-4.8Opus and GPT-5.4) never used in training.

Reviewer_3ctUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a research ideation method: using three agents and each one does one of the three tasks: identifying research gaps and the challenge, developing research ideas from analogous problem settings, and writing a research proposal. Each agent is trained with GRPO. The training rewards and evaluations are based on an llm judge on dimensions like novelty and feasibility. The authors ran a human evaluation for validating the evaluation judge. The main contributions are this research ideation pipeline, the dataset constructed from arxiv papers, and experimental results comparing a variety of models.

Critical strengths

The paper shows that the harness itself improves the performance (on novelty in particular) over various autoresearch baselines. The authors conducted evaluations using different backbone models and show consistent improvement across backbone models. Additionally, the authors conducted sub-agent ablations that show that adding the gap finder and innovator improves novelty. The thorough comparisons and ablations highlight the value of this harness. The second strength is that this work addresses a practically useful problem, which is how to generate high-quality and impactful research proposals, and provides a method that uses published papers as training signals. Research ideation lacks verifiable rewards, and this work provides some insights into how to construct scalable training signals.

Critical weaknesses

My first main concern is that the evaluation judges are not sufficiently validated, and novelty can trade off against feasibility and effectiveness. The human evaluation involves two annotators and 50 papers. The paper does not describe annotator expertise. Additionally, you could maximize novelty by drawing ideas from irrelevant domains, but you may not end up with feasible or effective ideas. The paper could be improved by adding evaluations that assess that tradeoff or adding other orthogonal eval metrics that are not as subjective as novelty.

My second main concern is that the gains from RL are small and probably not significant (novelty avg 66.9-> 68.8, quality avg 73.5 -> 74.6), but the paper's framing implies that RL is helpful. The main results table is missing confidence intervals.

Questions for authors
  1. Could you add more human evaluations on the evaluation metrics and more evidence (could be quantative or aulitative) showing that using this ideaScientist harness actually leads to useful research ideas beyond just increasing perceived novelty?
  2. Separating research ideation from AutoResearch is not new so should not be framed as one of your main contributions. What are the main differences between your work and these prior works:
  • Fan, Tianyu, et al. "Deepinnovator: Triggering the innovative capabilities of llms." arXiv preprint arXiv:2602.18920 (2026).
  • Ajith, Anirudh, et al. "PreScience: A Dataset and Benchmark for Scientific Forecasting." arXiv preprint arXiv:2602.20459 (2026).
  • He-Yueya, Joy, et al. "GIANTS: Generative Insight Anticipation from Scientific Literature." arXiv preprint arXiv:2604.09793 (2026).
9 comments

We thank the reviewer for recognizing the consistent gains across backbones, especially in novelty, the thorough comparisons and ablations, and the scalable training signal for non-verifiable ideation! We address each concern below.

W1/Q1. The evaluation judges are not sufficiently validated. The human study uses two annotators on 50 papers and does not describe their expertise.

  1. We will add details about our human annotation setup to the next version of the paper, described as follows.

Our two annotators are AI PhD students based in the US, both proficient in English. Both annotators have substantial reviewing experience at leading ML conferences (ICLR, NeurIPS, ICML, and COLM), each having reviewed more than 20 papers. Each annotator evaluated 50 random research problems, and for each problem, they evaluated the nine system proposals shown in Figure 2(b), all generated using the Qwen3.6-27B backbone. We built a dedicated web interface for blind annotation. Each screen displays the research challenge and problem definition, followed by one anonymous proposal organized into sections such as the method and evaluation plan. Each annotator uses a separate session with individually saved responses and can resume from their saved progress.

The scoring panel displays a definition for each of seven dimensions, soundness, clarity, specificity, actionability, impact, relevance, and novelty, and a shared 0–10 scale with explicit anchors: 0 = fundamentally inadequate, 5 = adequate, and 10 = exceptional. Annotators select an integer score for every dimension.

As reported in Section 5.3 (Figure 4a), we use these human annotations only to validate the trustworthiness of the LLM judges, not to train or select any system. Scoring research proposals is inherently subjective, yet we observe moderate correlations: the two experts’ scores correlate at Pearson r = 0.56, while the three best LLM judges correlate with the experts’ average scores at r = 0.51 to 0.59. The LLM judges are therefore a reasonable proxy for expert assessment.

  1. We agree that novelty and feasibility/effectiveness are indeed different dimensions. Ideally, feasibility and effectiveness shall be tested by implementing the method, conducting experiments, with iterations; but this is out of scope for this paper. Instead, Table 1 measures them with two separate sets of metrics: novelty against three scopes of literature (all literature, pre-cutoff literature and in-domain literature), and six review-style dimensions of proposal quality (relevance, clarity, specificity, actionability, soundness and impact), of which soundness and actionability directly capture feasibility.

Table 1 shows that IdeaScientist improves novelty substantially while also obtaining the best proposal quality. Additionally, the cross-judge evaluation in Section 5.3 (Figure 4) scored every proposal of IdeaScientist and of the seven open baselines with two frontier judges from other model families, Claude-4.8-Opus and GPT-5.4, which are never used anywhere in training. The paper reported only the agreement between judges, and we further show the per-baseline results behind it in the table below. Each entry is the score of IdeaScientist minus the score of the baseline under the same judge, so a positive value means IdeaScientist is better. IdeaScientist scores higher than almost every baseline not only in novelty but also in soundness and actionability:

Δ\Delta(IdeaScientist - Baseline)Claude-4.8-Opus: NoveltyClaude-4.8-Opus: SoundnessClaude-4.8-Opus: ActionabilityGPT-5.4: NoveltyGPT-5.4: SoundnessGPT-5.4: Actionability
AutoResearchClaw+17.5+8.9+10.0+11.3+3.2+3.0
Dr. Claw+18.3+3.8+5.9+12.4+2.0+4.4
ResearchAgent+21.8+5.7+7.2+15.0+1.2+2.4
CycleResearcher+18.0+3.3+5.0+10.0+0.7+2.0
SciAgents+21.8+6.2+7.5+14.6+1.9+2.5
AI-Researcher+13.5+4.3+6.1+5.3+1.0+0.8
DeepScientist+12.7+1.6+1.2+5.1+1.9-0.8
  1. To address whether the ideas are useful beyond perceived novelty, we added a new evaluation during the rebuttal: future-validated transfer. It counts a proposal as a future-validated transfer only if both of the following hold: (a) the proposal imports a mechanism from another field, as judged by Qwen3.6-27B, and (b) the proposal is more similar to the held-out target human paper, which was published after the cutoff and is invisible to the agent, than every pre-cutoff paper in the 4.7M-paper corpus is, measured by Qwen3-Embedding-0.6B cosine similarity. It means that the proposal anticipates the direction the field actually took better than any existing paper does, so an implausible idea borrowed from an irrelevant field would fail to satisfy it. The table reports the percentage of each system's proposals that satisfy both conditions; higher is better:
SystemProposals with a future-validated transfer
IdeaScientist28.5%
AutoResearchClaw10.8%
Dr. Claw15.3%
ResearchAgent10.0%
CycleResearcher8.2%
SciAgents7.8%
AI-Researcher14.3%
DeepScientist13.1%

W2. The gains from RL appear small.

Thank you for the question! We would like to clarify that RL brings three major benefits. First, RL improves what it directly optimizes, the agreement with the human-written research solution, and this improvement holds under all five judges, including the two frontier judges never used in training (paired differences over the test papers with 95% bootstrap confidence intervals):

JudgeTrained minus Base in reference agreement
Qwen3.6-27B (training judge)+0.054 [+0.037, +0.071]
Claude-4.8-Opus+0.077 [+0.054, +0.102]
GPT-5.4+0.072 [+0.055, +0.090]
Gemma-4-31B+0.106 [+0.084, +0.128]
GPT-OSS-120B+0.055 [+0.033, +0.078]

Second, training makes the full system more efficient: the trained agents read 26% fewer papers and make 15% fewer tool calls per proposal while preserving the iterative exploration and revision loops.

Third, these gains originate in the trained roles: each role improves significantly on held-out validation instances, by +10.6 [+6.9, +14.2] for the gap finder, +9.2 [+3.6, +14.6] for the innovator and +9.0 [+6.9, +11.4] for the report writer, and the improvement carries over to the final proposals through the higher reference agreement above. We will clarify these points in the next version of the paper.

Q2. How does IdeaScientist differ from DeepInnovator, PreScience, and GIANTS?

Thank you for raising this point. We view separating research ideation from execution as a motivation for our work rather than a contribution, and we will clarify this in the next version of the paper. We will cite the new references and below we describe the differences.

First, each of the three works studies a narrower version of the ideation problem:

(1) GIANTS reconstructs a paper's core insight in short sentences, from summaries of two oracle parent papers that the target paper is known to combine. It explicitly assumes an oracle literature selection and leaves retrieval of the relevant prior work to future work.

(2) DeepInnovator trains a single policy to write an idea from a context built offline from the target paper's own reference list, so it performs no literature search at inference time, and its cross-domain signal is mined only within those given references.

(3) PreScience is a forecasting benchmark rather than a method. It asks a model to predict a future paper's title and abstract from that paper's references, and scores how closely the prediction reproduces the realized paper. In all three settings, the model is told which prior work the target paper builds on, which removes a central part of research ideation, namely deciding what to build on, and the outputs are short sentences about research insights.

Our work addresses the full, open-ended problem: IdeaScientist receives only a problem and a challenge, must find what to build on by searching the literature itself, and must write a complete proposal with a method, an evaluation plan and falsifiable predictions. We build (1) a decomposition of ideation into subagents, coordinated by an orchestrator, each trained with GRPO against a stage-matched human reference, with an innovator that searches cross-domain mechanisms that solved an analogous challenge; (2) the Svalbard Idea Vault, which contains 2.77M stage-level tuples to provide this supervision. We hope this clarifies our contribution!

We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!

Reviewer_jqbWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents IdeaScientist, an orchestrated agent framework for finding research gaps, transferring solution intuitions across fields, and producing research proposals, with role-specific reinforcement learning. It constructs the Svalbard Idea Vault with 2.77 million decomposed research ideas and 14,977 target papers split into training, validation, and test sets. Evaluation measures novelty and proposal quality on 277 held-out tasks with retrieval restricted to pre-cutoff literature, supplemented by a separate human comparison on 50 tasks. The trained Qwen3.6-27B system scores 74.6 overall, versus 60.6 for the strongest matched-backbone baseline.

Critical strengths
  1. Assigning gap finding, intuition development, and proposal writing to specialized, orchestrated agents is a sensible decomposition of scientific ideation. The main-agent/sub-agent harness maintains a research plan while allowing specialist stages to be revisited, supporting extended exploration and refinement.

  2. The large-scale Idea Vault and its structured decomposition provide a substantial resource for studying scientific ideation.

  3. IdeaScientist-Base achieves higher overall scores than matched-backbone baselines on all three evaluated open models, supporting the portability of the harness.

Critical weaknesses
  1. Although the paper reports disjoint splits, training and evaluation share a source corpus and construction pipeline, leaving generalization beyond this setting unclear. Independently constructed ideation benchmarks, or compatible subtask evaluations such as idea generation and novelty assessment, would strengthen the evidence.

  2. Appendices F-G provide gap-finder rollout statistics and a compute-budget sweep, but do not give a full-system trajectory breakdown on the main test set. Per-role invocation/reinvocation distributions, step counts, action frequencies, and orchestrator turn statistics would help establish how the harness uses iterative exploration and refinement.

Questions for authors
  1. What evidence supports the inference in Section 3.2 that the orchestrator's narrow action vocabulary leaves little room for improvement? Keeping it fixed is understandable given training costs, but comparisons with alternative planning/scheduling policies under the same sub-agents, tools, and compute budget would test this claim more directly; expanding its action space could be a complementary ablation.

  2. How does role-specific GRPO compare with rejection-sampling fine-tuning on complete tool-use trajectories selected by the same reward? The sub-agents still learn multi-turn search and revision policies, so a strong sampling-and-SFT baseline, ideally with iterative data collection and matched rollout, judging, and training budgets, would clarify RL's added value.

8 comments

We thank the reviewer for recognizing the sensible role decomposition, the Idea Vault resource, and the portability of the harness across backbones! We address each point below.

W1. Training and evaluation share one source corpus and construction pipeline, which leaves generalization beyond this setting unclear.

Thank you for the question! We believe the benchmark can test generalization to unseen research directions for three reasons. First, we collected 14,979 arXiv papers spanning 93 primary categories across computer science, physics, mathematics, statistics, and electrical engineering, providing broad coverage of cutting-edge scientific research, as shown in the table below. Second, we manually reviewed all test papers to make sure they are diverse and differ from one another. We confirmed this quantitatively by embedding each paper’s decomposed fields with Qwen3-Embedding-0.6B and comparing cosine similarities. A test paper is no closer to its most similar other test paper than to its most similar paper in a random sample of the same size from our corpus (median similarity 0.66 vs. 0.67 for the key intuition, and 0.76 vs. 0.77 for the problem definition). Third, we deduplicated the training and test targets with both LLM-based and manual checking, so that no test paper overlaps with a training paper.

Primary arXiv categoryAll targets (14,979)Test papers (277)
Computer vision (cs.CV)23.4%24.2%
Machine learning (cs.LG)22.4%21.3%
Computation and language (cs.CL)12.9%14.1%
Artificial intelligence (cs.AI)9.5%8.7%
Robotics (cs.RO)4.6%6.5%
Information retrieval (cs.IR)2.5%2.5%
Cryptography and security (cs.CR)1.9%1.8%
Other computer science4.4%5.4%
Statistics (stat)1.0%0.7%
Electrical engineering (eess)1.2%1.1%
Mathematics (math)1.6%0.7%
Quantum physics (quant-ph)5.0%4.7%
High-energy physics (hep)4.1%3.2%
Astrophysics and gravitation (astro-ph, gr-qc)1.7%1.1%
Others3.7%4.0%

In addition, the data of existing ideation benchmarks are largely a subset of ours. Matched by arXiv identifier, our corpus contains 99.9% of the 17,196 GiantsBench target papers and 99.9% of the 97,819 PreScience target papers (99.7% of all 501,859 papers PreScience uses). These benchmarks therefore draw on a subset of the papers and fields covered by our data.

W2. A full-system trajectory breakdown on the main test set is needed to show how the harness uses iterative exploration and refinement.

We show the full-system trajectory statistics across the 277 test papers below. For distributions we report the mean, the median in parentheses, and the 10th–90th percentile range in brackets. All other entries are means or percentages of runs.

StatisticBaseTrained
Orchestrator turns34.9 (29) [14.6, 65.4]34.5 (27) [14.6, 69.0]
Sub-agent invocations11.0 (9) [5, 20]10.9 (9) [5, 23]
Gap finder invocations2.12.2
Innovator invocations3.73.7
Reviewer invocations3.63.4
Candidate intuitions generated3.73.7
Candidates not accepted by the reviewer2.62.5
Revision cycles (reviewer feedback followed by re-ideation)2.42.2
Papers read37.127.5
Tool calls199170
Runs completed in a single linear pass7.9%7.9%
Runs with at least one revision cycle73.3%67.9%
Runs returning to the gap finder after ideation began12.6%13.7%

The harness is used iteratively, where about four in five runs explore several gap axes, roughly 70% contain reviewer-driven revision loops, and the orchestrator goes back to earlier stages in a large percentage of runs. Training preserves this loop structure while making the agents much more efficient: the trained roles read only the papers they need and make more targeted tool calls, using 26% fewer paper reads and 15% fewer tool calls per proposal.

Q1. What evidence supports the claim that the orchestrator's narrow action vocabulary leaves little room for improvement?

As Appendix A.1 explains, the orchestrator neither retrieves literature nor reads papers. It takes only four actions: update the plan, read the artifacts generated by the subagents, invoke one of four specialist agents, or end the run once the final proposal exists. It averages 11 agent invocations per run and generates just 8% of output tokens. We therefore focus training on the specialist agents, which produce most of the content and learning signals.

Q2. How does role-specific GRPO compare with rejection-sampling fine-tuning under matched budgets?

We implemented the suggested baseline for all three producer roles. For each role, using the initial rollout batches from its GRPO run, generated by the base policy, we selected the highest-reward trajectory from each group of eight under the same reward, and fine-tuned Qwen3.6-27B on these trajectories for two epochs. We evaluated the base policy, rejection-sampling fine-tuning (RFT) and GRPO on the same held-out validation instances per role, using the same tools, paper reader, and judge:

RoleBaseRFTGRPO (ours)
Gap finder52.757.566.4
Innovator51.454.068.2
Report writer77.981.188.2

GRPO outperforms RFT by a large margin, and the gap reflects what on-policy RL adds: GRPO keeps exploring and improving with continued training, which offline selection from base-policy samples cannot provide without iterating data collection at the same scale. We will add these experimental results in the next version of our paper.

We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!

Comments on the paper

11 comments
~Zonglin_Yang1OpenReview ↗

Closely related prior work on decomposing and training research ideation

I am posting this comment because the submission's framing and architecture are closely related to a line of prior work ([1–4] below) that it does not currently cite. Relative to [1–4], the submission's new elements, as I read it, are role-specific RL training under reference-grounded rewards, the cross-domain vs. same-domain retrieval ablation (which shows a substantial novelty gain from retrieving outside the target domain), a proposal-format output that includes evaluation plans and falsifiable predictions, and the comparison with Claude Code / Codex SDK agents; these are useful additions. The points below concern the decomposition, the training-on-decomposed-roles idea, the data construction, and the "standalone task" claim.

1. Architecture. MOOSE-Chem [2] decomposed hypothesis discovery into inspiration retrieval → hypothesis composition → ranking, and showed that hypotheses in its benchmark can be rediscovered by composing a research background with inspirations retrieved from unrelated papers. MOOSE-Star [3] added a motivation-planning step ahead of retrieval. The submission's gap finder → innovator → report writer closely parallels this motivation planning → inspiration retrieval → hypothesis composition pipeline, including the premise that inspirations should be sought in other fields ([3] opens with the calculus → backpropagation example). This lineage is not currently discussed.

2. Training on decomposed roles. [3] trains separate inspiration retrieval and hypothesis composition models on the decomposed sub-tasks, shows that the decomposition is what makes training tractable (brute-force sampling pass rates fall to 0.23% at three required inspirations, Table 5), and reports, to our knowledge for the first time, training-scaling curves for scientific hypothesis generation (§7.2). The submission’s RL-based role training is a natural next step from SFT on the same decomposition; stating this relation would make clear what the RL training adds.

3. Decomposed-paper data. The Svalbard Idea Vault's (problem, challenge, gap, intuition, proposal) decomposition is close in design to TOMATO-Star, released with [3] (~109K papers decomposed into background / motivations / inspirations / per-inspiration delta-hypotheses, temporally split; publicly available), and to ResearchBench [4], which auto-decomposes papers into the same sub-tasks for evaluation. The Vault's scale and ML-domain focus are clear differences; it would help to state what else differs.

4. "We isolate research ideation as a standalone task" (Abstract). Standalone ideation has been studied since at least MOOSE [1], which showed LLM-generated hypotheses judged novel and valid by domain experts; SciMON and Si et al. (2025), both cited in the submission, and ResearchBench [4] likewise treat ideation as a standalone task. I take the intended point to be that ideation should be evaluated separately from the implementation stages of autoresearch systems, which is reasonable; but since ideation has been studied on its own for some time, the current wording could be read as claiming more than intended.

Beyond the points above, I think the RL training results and the cross-domain ablation are a useful step for this direction, and I am glad to see it being pushed further. I hope the comment is helpful for clarifying the submission's contribution relative to prior work; happy to discuss, and to be corrected if I have misread anything.

[1] Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. Findings of ACL 2024.

[2] MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses. ICLR 2025.

[3] MOOSE-Star: Unlocking Tractable Training for Scientific Discovery by Breaking the Complexity Barrier. ICML 2026.

[4] ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition. Findings of ACL 2026.

We thank Zonglin Yang for the thoughtful comment and for recognizing our role specific RL training, retrieval ablation across domains, proposal format, and comparison with frontier coding agents as useful additions!! We also appreciate you bringing these four papers you authored to our attention. We view separating research ideation from execution as a motivation for our work rather than a contribution, and we will clarify this in the next version of the paper. We will cite the new references and below we describe the differences:

(1) MOOSE-Chem and MOOSE-Star share the key constraint of GIANTS and PreScience, which Reviewer 3ctU also raised: the model is told which prior work the target paper builds on. GIANTS provides the two parent papers that the target paper combines, and PreScience provides its influential references; likewise, MOOSE-Chem and MOOSE-Star provide the target paper's research question and a survey of its background, and ask the model to select, from a provided candidate set, the inspirations that the target paper itself cites. All of them are then evaluated on how closely the output recovers the realized paper.

(2) ResearchBench, like PreScience, is a benchmark for this setting. IdeaScientist instead studies an open ended setting. Given only a problem and a challenge, it searches 4.7M arXiv papers published before the cutoff to identify relevant foundations and produces a full research proposal. The evaluation design of IdeaScientist considers both novelty relative to the literature and proposal quality, allowing valid alternatives to the realized paper to receive credit. Our Idea Vault supports this setting across all of arXiv with 2.77M stage-level tuples, while TOMATO-Star covers 109K biomedical papers, reflecting differences in both domain and scale.

Our experiments further examine two methodological differences. First, our innovator searches other fields for mechanisms that address analogous challenges, rather than relying on the target paper’s citations. This retrieval across domains improves novelty against all literature (+5.3) and against literature published before the cutoff (+3.7). Second, while MOOSE-Star uses rejection sampling fine tuning for individual subtasks, we train agents through multiple turns of tool use with on policy GRPO. In a controlled comparison using the same reward for each role, GRPO outperforms rejection sampling fine tuning on held out validation reward for every role (gap finder 66.4 vs. 57.5, innovator 68.2 vs. 54.0, report writer 88.2 vs. 81.1). Please see our response to Reviewer jqbW (Q2) for more details!

Thank you for the detailed and thoughtful reply, and for engaging with the comment. I am glad it was helpful, and that the relation to this line of work will be discussed in the next version.

One small clarification, in case it helps with the description there. In MOOSE-Chem and MOOSE-Star, the model is not told which prior work the target paper builds on. In MOOSE-Star, the ground-truth inspirations do come from the target paper's citations, but the model does not select only among those citations. At each retrieval step during deployment it searches a paper corpus (about 3K papers in our experiments; the method is not tied to this size) in which one paper is the ground truth and the rest have no particular relation to the target paper. The ground truth serves only as the label; to the model, it is one candidate among the others, with nothing marking it. On domain coverage, TOMATO-Star spans biology, medicine, computational imaging, and cognitive science, and ResearchBench covers 12 disciplines.

With that in mind, I would describe the relation as follows. In method, the main difference is the on-policy GRPO training over multi-turn tool use with reference-grounded rewards, together with the explicit search across fields and its cross-domain vs. same-domain ablation; these are useful additions. In setting, the differences are scale (your 4.7M-paper search space is far larger) and, relative to MOOSE-Chem and MOOSE-Star, evaluation (scoring novelty and proposal quality in addition to agreement with the realized paper; MOOSE also assessed novelty and validity, with expert evaluation). What the two lines of work share is a closely parallel decomposition (motivation/gap → inspiration retrieval → composition), which I think is worth noting alongside these differences.

I am glad to see this direction being pushed forward, and I appreciate the constructive exchange. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.