- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats research ideation as a standalone task: given a problem definition and a challenge, produce a research proposal. IdeaScientist uses an orchestrator and three sub-agents. A gap finder identifies a shared limitation in related work, an innovator looks for mechanisms from other fields that could transfer, and a report writer turns the result into a full proposal. Each role is trained separately with LoRA and asynchronous GRPO, using a reward that combines a reference-free LLM score, an LLM comparison against a reference built from the human paper, and citation F1. The authors also build the Svalbard Idea Vault, 2.77M ideas decomposed from 4.7M arXiv papers. They use papers after a January 2026 cutoff as training and test targets, and retrieval at evaluation time is restricted to pre-cutoff literature. On Qwen3.6-27B, the trained system beats the best open autoresearch baseline by 14% and also scores above Claude Code (Claude-4.8-Opus) and Codex (GPT-5.4), mainly because of higher novelty scores.
Critical strengths
- Studying ideation on its own makes sense. With end-to-end systems it really is hard to tell whether gains come from the idea or from the execution.
- Training each role on its own reference artifact is a reasonable way around the sparse reward of the full pipeline.
- The paper includes some analysis of judge reliability (cross-judge agreement, self-preference) and a human pairwise study.
Critical weaknesses
- The training reward conflicts with the novelty goal. r_ref rewards agreement with a Qwen-written reconstruction of the target paper, so the agents are trained to reproduce the human paper’s solution. This sits uneasily with the paper’s main claim that IdeaScientist’s advantage comes from novelty. It would help to explain how the reward avoids penalizing valid but different solutions, and to check whether r_ref agrees with human judgments of quality.
- The same judge is used for training and evaluation. Qwen3.6-27B builds the references, gives the GRPO reward, and is the main judge in Table 1, so part of the gain is expected by construction.
- The risk of contamination and leakage is underestimated. The targets come from January–June 2026, while Qwen3.6, Claude-4.8, and GPT-5.4 were released between March and May 2026. A title-only probe on ten papers is weak. Challenges rewritten from the full paper may already point toward its solution, and the random split within one six-month window likely puts closely related papers in both train and test.
- The ablations are limited. The sub-agent ablation reports only novelty. The retrieval ablation’s main gain is on the In-domain metric, which is biased toward cross-domain transfer to begin with. There’s no RL vs. SFT comparison.
Questions for authors
- Without the In-domain column, does Overall still favor IdeaScientist over Claude Code and Codex?
- With Claude or GPT as the main judge, does Trained still beat Base?
- How well do the training reward and its components agree with human judgments? Is there an ablation over the components and weights?
10 comments
W3. The evaluated models may have memorized post-cutoff papers, and training and test targets may leak into each other.
We consider January 1, 2026 a reasonable cutoff that balances model capability with contamination risk for three reasons. First, GPT-OSS-120B was released on 2025-08-05, before our 2026-01-01 cutoff, so it cannot have seen any post-cutoff target paper during training. Second, as reported in Appendix E.2, we manually queried our evaluated models about sampled post-cutoff target papers, providing only the paper title without retrieval, and none of the models identified or described any of them as known work.
For leakage between training and test targets, we manually reviewed all test topics and deduplicated the training and test targets using both LLM-based and manual checks to ensure that no test paper overlaps with a training paper.
W4. The ablations are limited: the sub-agent ablation reports only novelty, the retrieval ablation's gain is mainly In-domain, and there is no RL vs. SFT comparison.
Thank you for the question! We extend our ablation studies to the other judge dimensions. The table below reports all six quality dimensions, their average, and novelty against all literature for the test papers, using the two backbones in the ablation without training:
| Backbone | Configuration | Relevance | Clarity | Specificity | Actionability | Soundness | Impact | Average quality | Novelty against all literature |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3.6-27B | Report writer only | 95.5 | 85.4 | 81.9 | 75.6 | 59.2 | 70.7 | 78.0 | 59.5 |
| Gap finder and report writer | 97.2 | 90.1 | 86.4 | 78.7 | 64.2 | 75.3 | 82.0 | 63.4 | |
| Full pipeline (with innovator) | 96.2 | 89.9 | 80.6 | 77.3 | 62.8 | 73.4 | 80.0 | 66.0 | |
| Gemma-4-31B | Report writer only | 95.1 | 84.2 | 71.9 | 70.2 | 57.8 | 67.1 | 74.4 | 49.6 |
| Gap finder and report writer | 94.5 | 88.0 | 73.5 | 72.2 | 60.1 | 70.6 | 76.5 | 55.8 | |
| Full pipeline (with innovator) | 94.9 | 89.1 | 74.5 | 73.3 | 60.3 | 70.6 | 77.1 | 61.4 |
We can see that the gap finder and report writer together contribute mostly to the other dimensions, while the innovator provides a strong boost in novelty. The other judges are not affected much by whether the innovator is included, because they care mostly about the quality of the writing and presentation without judging the novelty of the idea itself.
We also ran the requested RL vs. SFT comparison for all three producer roles, fine-tuning Qwen3.6-27B on the best of eight base-policy trajectories per group (rejection-sampling fine-tuning, RFT) and evaluating all policies on the same held-out validation instances with the same tools and judge. We found that GRPO outperforms RFT on every role by a large margin:
| Role | Base | RFT | GRPO (ours) |
|---|---|---|---|
| Gap finder | 52.7 | 57.5 | 66.4 |
| Innovator | 51.4 | 54.0 | 68.2 |
| Report writer | 77.9 | 81.1 | 88.2 |
Q1. Without the In-domain column, does Overall still favor IdeaScientist over Claude Code and Codex?
Without the In-domain column, IdeaScientist remains the best system on every open backbone (74.1 vs. 67.8 on Qwen3.6-27B, 69.5 vs. 64.2 on Gemma-4-31B, and 67.8 vs. 64.3 on GPT-OSS-120B). On matched frontier backbones, it outperforms Claude Code on Claude-4.8 (81.5 vs. 77.4), while Codex scores higher on GPT-5.4 (78.3 vs. 75.2). We recompute Overall from the submitted Table 1 as the mean of proposal-quality Avg and the mean of All and Cutoff novelty. IdeaScientist uses Trained on Qwen3.6-27B and Base on the other backbones:
| Backbone | IdeaScientist | Best other system |
|---|---|---|
| Qwen3.6-27B | 74.1 | 67.8 (DeepScientist) |
| Gemma-4-31B | 69.5 | 64.2 (AI-Researcher) |
| GPT-OSS-120B | 67.8 | 64.3 (AI-Researcher) |
| Claude-4.8 | 81.5 | 77.4 (Claude Code) |
| GPT-5.4 | 75.2 | 78.3 (Codex) |
Q3. How well do the training reward and its components agree with human judgments, and were the components and weights ablated?
We found that the reward agrees with human judgments, mainly through its reference-free component. We matched human-annotated proposals with valid stored report-writer rewards and computed proposal-level Pearson correlations with the two annotators’ mean scores, using paper-clustered 95% confidence intervals:
| Reward signal | Correlation with human quality (six dimensions) | Correlation with human novelty |
|---|---|---|
| Reference-free quality | 0.52 [0.44, 0.60] | 0.36 [0.27, 0.44] |
| Reference agreement | 0.07 [−0.06, 0.19] | −0.01 [−0.15, 0.12] |
| Citation F1 | 0.14 [0.03, 0.24] | 0.19 [0.08, 0.28] |
| Full report-writer reward | 0.33 [0.21, 0.43] | 0.24 [0.12, 0.35] |
The components play complementary roles. The reference-free component carries the agreement with human judgments of quality (0.52) and novelty (0.36), while reference agreement measures a different property, agreement with the research direction that the field actually took, and is neither associated with human quality nor with human novelty, so it does not penalize novel proposals. Its effect is instead visible in what training improves: agreement with the held-out human solution rises significantly under all five judges, including two frontier judges (Claude-4.8Opus and GPT-5.4) never used in training.
We thank the reviewer for recognizing the motivation for studying ideation on its own, the stage-matched training signal for sparse rewards, and the judge-reliability analysis! We address each concern below.
Thank you for the question! The report-writer reward consists of four components: a reference-free quality score that judges the proposal on its own, a reference-grounded score that compares the proposal with the human-written solution, a citation score, and a process term penalizing protocol violation. The reference-grounded component therefore contributes only 28% of the training reward.
The reference-grounded component is designed to reward substantive research decisions: every target paper in our data is a methodology paper that proposes a new method, so the reference-grounded component rewards the kind of substantive methodological decisions that such papers make, rather than surface similarity. We agree that such a reward could in principle penalize a different but equally novel method. The reward definition avoids this with the use of reference-free component rewarding the standalone quality of every proposal independently of the reference.
Empirically, we find no negative association between reference agreement and novelty, here measured as the judge's novelty against all literature, and for human annotations as human-rated novelty:
All correlations are close to zero and none is significantly negative, so proposals that agree more with the human reference are not less novel, and training that raised reference agreement did not lower novelty on the same papers.
Finally, we pointed out that similarity to the realized paper is also the main training or evaluation signal in recent ideation work such as GIANTS and DeepInnovator, as noted by Reviewer 3ctU and in Zonglin Yang’s public comment. We improved by defining reference-free evaluation rubrics.
Thank you for this question. We would like to clarify that the improvement from training is not tied to the training judge. The cross-judge evaluation in Section 5.3 (Figure 4) scored every IdeaScientist proposal with Claude-4.8-Opus and GPT-5.4, which are never used anywhere in training, as well as Gemma-4-31B and GPT-OSS-120B. Since the paper reported only the agreement between judges, we additionally report here the training effect under each judge. Agreement with the research direction of the held-out human paper, the quantity that RL optimizes, improves under every judge:
Under both frontier judges, Trained also improves average proposal quality over Base (68.8 vs. 70.0 under Claude-4.8-Opus and 74.5 vs. 76.0 under GPT-5.4). In addition, the harness comparisons in Table 1 also hold under the frontier judges: IdeaScientist-Base exceeds every one of the seven open baselines in novelty by +12.7 to +21.8 points under Claude-4.8-Opus and by +5.1 to +15.0 points under GPT-5.4.