- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies document-grounded scientific ideation: given summaries of several source papers, a model must write K = 3 structured idea cards (problem, claim, method, inspiration) that are each high-quality and mutually distinct. Training uses a set-level reward adapted from Creative Utility, in which each idea’s quality is multiplied by its minimum distance to the ideas written earlier in the same generation. Quality is a pairwise impact judgment against the held-out “gold” idea of the target paper, made by SciJudge-30B; distance is an LLM-judged field-wise similarity from gpt-oss-120b. Qwen3-8B is first fine-tuned on Claude-generated idea sets with hindsight reasoning traces and then trained with GRPO. On a test set built from 2026 papers, the no-think model reaches a CU of 0.64, compared with 0.38 for Claude Opus 4.8 and 0.52 for a retrained DARLING baseline. A four-expert pilot study finds parity with Claude, and a 15-instance execution study with a coding agent finds comparable success@k.
Critical strengths
- A clean demonstration that in-generation, set-level diversity rewards avoid collapse. With the same base model, data and judges, impact-only RL collapses the set (distance 0.38), and DARLING collapses idea types on IdeaSeed. In contrast, the proposed objective keeps both quality and diversity high (CU 0.64, distance 0.88).
- External checks go beyond the training metric. The paper reports grounding, population-level type diversity, a blinded expert study and an execution study. Each is small, but together they suggest the gains are not purely reward hacking and that an 8B model reaches parity with Claude Opus 4.8 in expert judgment.
Critical weaknesses
- The sample-then-select baseline is missing. The most natural alternative is to draw N independent ideas from the SFT model or Claude and then select K with the same quality × distance greedy procedure used to compute CU. Without this baseline, it is unclear whether training a model to diversify within one generation is better than generating and then selecting.
- What is the inter-annotator agreement in the human study, and are any differences between Proposed and Claude statistically significant? How were the 60 instances selected?
- DARLING is trained with K = 1. How is it evaluated at K = 3? Would DARLING with the same set-format SFT be a fairer comparison?
Questions for authors
see above
3 comments
Thank you again for your thoughtful review. As the discussion period is nearing its end, we would like to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns. We would greatly appreciate any further feedback you may have, and we would be happy to clarify any remaining questions!
THank you for your response. I have increased the soundness score
Thank you for your feedback and review! Addressing the weaknesses and questions below:
In Table 12 we report set vs sampling performance. We see that set/single generation has higher diversity compared to sampling based. We also run a sample-then-select baseline. For Claude Opus 4.8, GPT 6 Astra and Qwen3-8B we sample 9 times, and use the CU greedy algorithm to pick the best 3. For scoring and selection we use the training judge and then re-score the selected set with our evaluation judges. Results:
For Claude, sample then select (or best of n) performs at par with set level generation, still performing lower than our trained models (0.64/0.51, Table 1). We want to note this sample-then-select method is an oracle selected method. Selection uses the target ideas, which are not available at inference time. We also want to mention that this baseline is more expensive than our propsoed method e.g. this uses 9 generations per instance instead of 1.
We present a human evaluation in Section 6.1 and Appendix D. In that each expert gives judgments on 15 independent instances, so the design has no shared items so an item-level IAA cannot be computed. We instead measure agreement at the system level. For each expert, they rank 15 instances across- three systems, and with four experiments, we get a Kendall's tau of 0.56 for "promising and unique ideas" (average pairwise kendall's tau) . All four experts agree on the base model's ranking (i.e. last). However, they split 2–2 between our model and Claude, consistent with the parity we report.
We report paired differences, 95% bootstrap CIs, bold means significant after Bonferroni correction over the 9 tests. We find no difference between our proposed system and Claude, which is still notable for an 8B model compared to a frontier model.
The 60 instances come from the NLP subset of our test set. Each expert chooses their area of expertise (e.g. agents, evaluation, bias and ethics etc). An LLM classifier tags each test paper with its ARR subarea. We then randomly sample 15 instances per expert that match their expertise. No step in this selection chooses ideas or instances based on their scores. Experts were also blind to the system name when giving judgments. We mention the details in Appendix D (line 1212) and will make it more clear.
DARLING generates one idea per call, as it was trained. For each test instance we sampled 3 single-idea rollouts and took that as the set of ideas generated from DARLING (Section 4). Since DARLING's diversity reward is computed across a group of independent samples, independent sampling is its intended way to produce several ideas.
We trained DARLING + SFT. We retrained DARLING from the same SFT (no-think) checkpoint that our method starts from (but with k = 1). We train it with a length penalty and for the same number of steps. We see the following:
Even starting with our SFT model does not close the gap for DARLING. While it raises the best idea per set (>=1 Win goes from 0.59 to 0.65), the sets DARLING + SFT generates become less distinct (distance 0.67 goes down to 0.62). Our no-think model still beats DARLING + SFT. We will add this result to the updated draft of the paper.