acceptodds
Under review as a conference paper at ICLR 2027

GIANTS: Generative Insight Anticipation from Scientific Literature

Abstract

Scientific breakthroughs often emerge from synthesizing prior ideas into novel contributions. While language models (LMs) show promise in scientific discovery, their ability to perform this targeted, literature-grounded synthesis remains underexplored. We introduce insight anticipation, a generation task in which a model predicts a downstream paper’s core insight from its foundational parent papers. To evaluate this capability, we develop GiantsBench, a benchmark of 17𝑘 examples across eight scientific domains, where each example consists of a set of parent papers paired with the core insight of a downstream paper. We evaluate models using an LM judge that scores similarity between generated and ground-truth insights, and show that these similarity scores correlate with human expert ratings. Finally, we present GIANTS-4B, an LM trained via reinforcement learning (RL) to optimize insight anticipation using these similarity scores as a proxy reward. Despite its smaller open-source architecture, GIANTS-4B outperforms proprietary baselines and generalizes to unseen domains, achieving a 33% relative improvement in similarity score over gemini-3-pro. Human evaluations further show that GIANTS-4B produces insights that are more conceptually clear than those of the base model. In addition, SciJudge-30B, a third-party model trained to compare research abstracts by likely citation impact, predicts that insights generated by GIANTS-4B are more likely to lead to higher citations, preferring them over the base model in 68% of pairwise comparisons. We release our code, benchmark, and model to support future research in literature-grounded insight anticipation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific insight anticipation, research idea synthesis, Literature-based synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_fRdcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces insight anticipation, a task that predicts a downstream paper's core insight from summaries of two parent papers. The evaluated direction is from parent papers to downstream insights. Identifying the parents is only part of dataset construction. To build the dataset, a model reads each downstream paper, selects two cited parents, and explains how their ideas are combined. This process produces GIANTSBENCH, with about 17,000 examples, making research idea generation measurable against published contributions.

Critical strengths

It offers a more testable question than open-ended brainstorming, i.e., given prior papers, can a model predict how later work combines them?

Critical weaknesses
  • The validation of parent-paper selection is limited. Only 50 papers are checked by external experts, whose judgments cannot establish the authors' actual sources of inspiration. And model agreement of multiple LLM judges would not establish true provenance. Collecting feedback from some original authors might strengthen the evidence, but I still have serious doubts about this research setting.

  • The construction prompt requires two parent papers without an explicit option to report that no suitable pair exists. The paper does not report failure rates, exclusion rules, or manual handling

  • The assumption of exactly two parent papers needs stronger support, especially when the ground truth is not necessarily correct. Explaining this choice by controlling the input length is not convincing.

Questions for authors

Which specific downstream application will this task (insight anticipation) improve? And which experiments in the current paper can demonstrate that it will indeed be helpful?

2 comments

We thank Reviewer fRdc for recognizing the verifiability of our setting. Below, we addressed the concerns you raised related to parent-paper selection, data construction, and downstream application of our task.

The validation of parent-paper selection is limited. Only 50 papers are checked by external experts, whose judgments cannot establish the authors' actual sources of inspiration [...].

Thank you for raising this. We agree that expert ratings and agreement between LLM judges cannot establish the authors' actual sources of inspiration. As the reviewer suggests, we therefore validated our parent selection against the original authors.

ScholarCatalyst [3] is a recently released dataset in which 184 authors labeled, for 207 of their own recent papers, which prior papers inspired the work and which plausible candidates did not. We ran our unchanged parent-identification step on all 207 papers: the same prompt, with the full PDF as input. One difference is the model: we used gemini-3.7-flash, because gemini-2.5-flash is no longer available to API users. Of the 414 selected parents, 72% are papers the authors themselves marked as inspirations, and only 4% were explicitly rejected by the authors. The remaining 24% were never shown to the authors, so they carry no label, and we count them as not confirmed. Among selected parents that the authors did judge, 95% were inspirations, compared with 44% for a randomly chosen paper from the same author-judged pool. 86% of pairs contain at least one author-confirmed parent, and 58% have both. ScholarCatalyst's own model-based reconstruction reaches a similar level of agreement with the authors: 58.2% precision for the main research question, versus 56.2% for our parents on the same criterion. This suggests that LLM selection from a paper's cited works recovers genuine inspirations far above chance, though not perfectly. We will add this analysis and its scope (computer science, papers from 2025 onward) to the paper.

On the research setting. GIANTS does not require the selected parents to be the complete or sole origin of the target paper. It requires that the target insight be predictable from them. Appendix E.2 tests this directly. Similarity falls from 4.90 with both true parents to 2.99 with one parent, 1.98 when a parent is replaced by an unrelated paper, and 1.00 with no parents. So the target insight is recoverable from the selected parents and not from a prior over target insights. Any remaining noise in parent selection is shared by all models, so it does not affect the comparison between them. Together with the author-level evidence above, we believe this supports the benchmark as a test of synthesizing insights from the literature that actually inspired the work.

The construction prompt requires two parent papers without an explicit option to report that no suitable pair exists. The paper does not report failure rates, exclusion rules, or manual handling

Thank you for pointing this out. We will add the full construction funnel and the exclusion rules to the appendix. We summarize both here, along with new evidence on how often a two-parent structure fits real papers.

We ran the parent-identification prompt on 519,380 arXiv papers. Two parent titles could be extracted for 503,761 (97.0%). The remaining 3.0% were excluded. These were API failures (0.1%) and, mostly, responses that named papers without the required output tags. Explicit statements that no suitable pair exists were very rare (39 papers, under 0.01%). We then applied fully automatic exclusion rules, with no manual handling: (1) Both parent titles must exactly match a published paper whose full text we could obtain and summarize. 112,803 papers remain. This step also removes hallucinated or garbled titles, which cannot match. (2) The target paper must have at least two citations. 110,373 papers remain. The temporal train/test split and per-domain sampling are then applied to these 84,895 papers.

How often do two parents fit? To measure how restrictive the two-parent assumption is, we used ScholarCatalyst [3], in which 184 authors labeled the key inspirations of 207 of their own papers. Most papers have at least two key inspirations, and only 13% have exactly one.

Author-labeled key inspirations12345 or more
Share of papers13.0%28.0%25.6%13.5%19.8%

The median is three, and 87% of papers have two or more. Forcing exactly two parents is therefore rarely asking for something that does not exist. For papers with more than two key inspirations, the selected pair is a subset of the paper's true foundations, which matches the task: the target insight only has to be predictable from the given parents. We also ran our pipeline on these papers. For the 13% with a single key inspiration, the forced second pick was rarely wrong: authors rejected only 6% of those picks.

Going beyond two parent papers for context purposes

We agree with the reviewers framing that two going beyond two parent papers can be addressed through alternative forms of context management. We now run an experiment, where we use a stronger LLM (DeepSeek-V4.1-Flash) agentically with the pi harness (pi.dev), a common harness used by prior work [1,2] for interacting with a filesystem. Here, we supply the agent with n randomly sampled papers extracted from the related works of a target paper in a flat filesystem that the model can access and browse via the harness. We then perform inference time scaling (best@k), to understand the how the similarity evolves with more insights that are generated. We see that there is a monotonic increase in performance as more reference papers are seen, indicating that this is helpful for prediction. The best@k performance also indicates that through simple sharpening with RL, we can get to this target insight. For the final version of the manuscript and the dataset release, we will setup a filesystem variant of the evaluation and fine-tune a model with the harness.

Reference papersbest@1best@2best@4best@8
12.533.033.483.84
22.983.403.764.06
43.453.824.084.28
83.483.894.204.47
163.714.084.394.62

Which specific downstream application will this task (insight anticipation) improve? And which experiments in the current paper can demonstrate that it will indeed be helpful?

Thank you for the question. The main downstream application is literature-grounded research ideation: given papers a researcher has selected, the model proposes a concrete insight that combines them. This is the synthesis step in human-in-the-loop research assistants and in automated research pipelines, which first retrieve relevant work and then propose ideas to pursue. Insight anticipation evaluates this step directly, because published downstream papers provide a grounded record of which combinations of prior work actually led to new research.

Several experiments in the paper indicate that GIANTS-4B would be helpful for this application:

  1. Temporal forecasting. All evaluations use downstream papers published after the training cutoff, including papers published after the base model's release (Appendix C). GIANTS-4B therefore anticipates insights that had not yet appeared in its training data, which is the setting a research assistant faces.

  2. Usefulness to researchers. In our human study, annotators who did not see the ground truth or the model identity rated GIANTS-4B's insights as significantly clearer and better scoped, i.e., more focused and testable, than those of the base model and Gemini-3-Pro, at comparable implementation complexity (Figure 5). These are the properties a researcher needs to act on an idea.

  3. Potential impact. SciJudge-30B, a model trained to predict citation impact, prefers GIANTS-4B's insights over the base model's in 68% of comparisons (Figure 4).

  4. Generate-then-select pipelines. GIANTS-4B remains the strongest model as the number of samples increases (Figure 2), so it can supply a stronger pool of candidates for a system that generates several ideas and then filters them.

  5. Breadth. Although trained only on cs.CL, GIANTS-4B improves in all 16 domains (Figure 3), so a single model can support researchers across fields.

We deliberately focus on insight generation given a fixed set of parent papers. Our goal is to evaluate whether models can generate novel yet tractable research insights when provided with rich contextual knowledge: specifically testing their ability to make conceptual leaps [4] and escape "idea basins" [5,6]. Utilizing citation graphs from published literature provides a structured warm-start objective for research agents, as human scientific progress relies on synthesizing existing work into meaningful new contributions.

References

[1] Learning in a World of Notes (https://engram.com/blog/learning-in-a-world-of-notes)

[2] Harness Leaderboard (https://huggingface.co/spaces/joelniklaus/harness-leaderboard)

[3] ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research (https://arxiv.org/abs/2610.02202)

[4] Position: LLMs can't jump (https://openreview.net/forum?id=klU4737opt)

[5] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents (https://arxiv.org/abs/2610.07447)

[6] How models explore ideas (https://bespokelabs.ai/blog/how-models-explore-ideas)

Reviewer_1Ds7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the GIANTS framework, designed to evaluate and enhance the ability of language models to generate scientific insights by synthesizing existing research. The authors propose a new task called "insight anticipation": given the abstracts of two foundational papers, a model must predict the core contributions of subsequent, related research. To support this task, the authors developed the GIANTS Bench dataset, which spans eight scientific fields and contains approximately 17,000 samples. They also propose an evaluation metric based on large language models (LLMs) to measure the semantic similarity between the insights generated by a model and the actual contributions of subsequent research. Furthermore, the authors developed the GIANTS-4B model (with 4 billion parameters), trained using the Group Relative Policy Optimization (GRPO) algorithm and a similarity-based reward mechanism. Experiments demonstrate that GIANTS-4B outperforms various proprietary and open-source baseline models; in the "unseen-parent" evaluation scenario, it shows a 33% performance improvement over Gemini-3-Pro.

Critical strengths

Despite its relatively small scale, the proposed GIANTS-4B demonstrates promising performance when compared to a variety of proprietary and open-source language models.

Critical weaknesses

In my point of view, the paper has several notable shortcomings:

  1. The benchmark primarily evaluates performance by calculating the semantic similarity between the generated insights and the actual contributions of subsequent research (downstream papers). Consequently, what the benchmark measures is less an ability to make genuine scientific discoveries and more an ability to "replicate historical research trajectories."

  2. The proposed training methodology offers limited innovation. GIANTS-4B employs GRPO; it introduces no new reinforcement learning algorithms, nor does it incorporate any specialized mechanisms for modeling conceptual relationships across different papers.

  3. Defining the true conceptual foundation of a scientific contribution—that is, identifying the "parent papers"—is an exceptionally difficult task, and this may well represent a significant flaw in the paper.

Questions for authors

None

3 comments

We thank Reviewer 1Ds7 for recognizing the performance of GIANTS-4B when compared to proprietary and open-sourced models. Below, we addressed the concerns you raised related to task framing, the usage of GRPO, and parent paper identification.

what the benchmark measures is less an ability to make genuine scientific discoveries and more an ability to "replicate historical research trajectories."

Thank you for this comment. We agree that GIANTSBench measures how well a model anticipates research directions that were actually pursued, not open-ended scientific discovery, and we will revise any wording in the paper that suggests otherwise. We see this as a deliberate and useful scope for three reasons.

First, realized downstream papers are the only ground truth for good ideas that is available at scale. Ideas that the community actually pursued and published are a verifiable subset of valuable ideas. Judging whether a new idea is a genuine discovery requires executing and validating it, which is costly and slow (e.g., [1]). Anticipating insights from the parent papers requires the same core skill that discovery relies on: synthesizing and building upon prior work to create a new contribution. This follows the logic of backtesting in forecasting, also used by PreScience [2]. A model that cannot recover directions that experts did pursue from the same inputs is unlikely to propose good new ones.

Second, the benchmark does not reward recalling a single historical trajectory. The same parent papers often led to several different downstream papers. Appendix D evaluates this directly: we sampled 30 parent pairs that are each shared by at least four downstream papers, and scored each generated insight against all of them. GIANTS-4B achieves the highest maximum similarity (6.46, vs. 5.42 for the strongest baseline, Qwen3-14B), as well as the highest median and mean.

ModelMaxMedianMean
Gemini 3 Pro4.46 ± 0.423.04 ± 0.163.11 ± 0.18
Qwen3-4B (base)5.08 ± 0.473.35 ± 0.273.52 ± 0.23
Qwen3-14B5.42 ± 0.463.58 ± 0.273.74 ± 0.25
GIANTS-4B6.46 ± 0.494.06 ± 0.314.31 ± 0.27

The ablations in Appendix E.2 also show that scores depend on reasoning from the given parents: similarity falls from 4.90 with both true parents to 2.99 with one parent, 1.98 when a parent is replaced by an unrelated paper, and 1.00 with no parents. Models therefore cannot score well by drawing on a prior over well-known research directions. Finally, the train/test split is temporal, so test insights come from papers published after the training papers.

We now run an experiment to showcase that the requirement of the best paper identified by gemini is perhaps not needed. In this experiment, we use a stronger LLM (DeepSeek-V4.1-Flash) agentically with the pi harness (pi.dev), a common harness used by prior work [4,5] for interacting with a filesystem. Here, we supply the agent with n randomly sampled papers extracted from the related works of a target paper in a flat filesystem that the model can access and browse via the harness. We then perform inference time scaling (best@k), to understand the how the similarity evolves with more insights that are generated. We see that there is a monotonic increase in performance as more reference papers are seen, indicating that this is helpful for prediction. The best@k performance also indicates that through simple sharpening with RL, we can get to this target insight. This showcases that an agent supplied with a wider range of reference papers and effective context management can still disentagle the target insight with inference time computation. For the final version of the manuscript and the dataset release, we will setup a filesystem variant of the evaluation and fine-tune a model with the harness.

Reference papersbest@1best@2best@4best@8
12.533.033.483.84
22.983.403.764.06
43.453.824.084.28
83.483.894.204.47
163.714.084.394.62

Third, we view insight anticipation as a warm-start for ideation, for example as a mid-training objective, rather than as a replacement for open-ended discovery. Open-ended discovery can then be grounded in execution, as in [1]. We will make this framing explicit in the introduction and limitations, and describe GIANTSBench as measuring literature-grounded insight anticipation.

GIANTS-4B employs GRPO; it introduces no new reinforcement learning algorithms

Thank you for the comment. We agree, and we do not claim a new RL algorithm, and directly use the policy gradient objective of GRPO. Our contributions instead are the task definition (insight anticipation from parent papers), the benchmark (GIANTSBench), and showing that this task formulation gives a scalable training signal for ideation: an LM judge's similarity to a realized downstream insight, which we validate against expert ratings. We use standard GRPO deliberately, so that the gains of GIANTS-4B can be attributed to the task and reward rather than to algorithmic changes. A 4B model trained this way outperforms much larger open models and frontier models such as gemini-3-pro on the benchmark. The setup is also algorithm-agnostic, so any improved RL method could be applied to it directly. We will state this scope more explicitly in the paper.

Defining the true conceptual foundation of a scientific contribution—that is, identifying the "parent papers"—is an exceptionally difficult task, and this may well represent a significant flaw in the paper.

Thank you for raising this. We agree that identifying the parent papers may be an important and difficult task. We therefore validated our parent selection against the original authors.

We deliberately focus on insight generation given a fixed set of parent papers. Our goal is to evaluate whether models can generate novel yet tractable research insights when provided with rich contextual knowledge: specifically testing their ability to make conceptual leaps [6] and escape "idea basins" [7,8]. Utilizing citation graphs from published literature provides a structured warm-start objective for research agents, as human scientific progress relies on synthesizing existing work into meaningful new contributions. Furthermore, upstream parent-identification tools such as ScholarCatalyst [3] and MIR [9] can directly complement our framework. An ideal system would leverage both the ability to identify key papers as well as learning how to synthesize them into a useful insight.

To study the validity of our current paper selection, we run an experiment on ScholarCatalyst [3], a recently released dataset in which 184 authors labeled, for 207 of their own recent papers, which prior papers inspired the work and which plausible candidates did not. We ran our unchanged parent-identification step on all 207 papers: the same prompt, with the full PDF as input. One difference is the model: we used gemini-3.7-flash, because gemini-2.5-flash is no longer available to API users. Of the 414 selected parents, 72% are papers the authors themselves marked as inspirations, and only 4% were explicitly rejected by the authors. The remaining 24% were never shown to the authors, so they carry no label, and we count them as not confirmed. Among selected parents that the authors did judge, 95% were inspirations, compared with 44% for a randomly chosen paper from the same author-judged pool. 86% of pairs contain at least one author-confirmed parent, and 58% have both. ScholarCatalyst's own model-based reconstruction reaches a similar level of agreement with the authors: 58.2% precision for the main research question, versus 56.2% for our parents on the same criterion. This suggests that LLM selection from a paper's cited works recovers genuine inspirations far above chance, though not perfectly. We will add this analysis and its scope (computer science, papers from 2025 onward) to the paper.

Additionally, GIANTS does not require the selected parents to be the complete or sole origin of the target paper. It requires that the target insight be predictable from them. Appendix E.2 tests this directly. Similarity falls from 4.90 with both true parents to 2.99 with one parent, 1.98 when a parent is replaced by an unrelated paper, and 1.00 with no parents. So the target insight is recoverable from the selected parents and not from prior over target insights. Any remaining noise in parent selection is shared by all models, so it does not affect the comparison between them. Together with the author-level evidence above, we believe this supports the benchmark as a test of synthesizing insights from the literature that actually inspired the work.

References

[1]The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas (https://arxiv.org/abs/2506.20803)

[2] PreScience: A Dataset and Benchmark for Scientific Forecasting (https://arxiv.org/abs/2602.20459)

[3] ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research (https://arxiv.org/abs/2610.02202)

[4] Learning in a World of Notes (https://engram.com/blog/learning-in-a-world-of-notes)

[5] Harness Leaderboard (https://huggingface.co/spaces/joelniklaus/harness-leaderboard)

[6] Position: LLMs can't jump (https://openreview.net/forum?id=klU4737opt)

[7] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents (https://arxiv.org/abs/2610.07447)

[8] How models explore ideas (https://bespokelabs.ai/blog/how-models-explore-ideas)

[9] MIR: Methodology Inspiration Retrieval for Scientific Research Problems (https://arxiv.org/abs/2506.00249)

Reviewer_TuTyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Proposes "insight anticipation": given two parent-paper summaries, predict a downstream paper's core insight (one realized y∗ among many plausible derivable insights from the pair). Builds GIANTBENCH, 17k examples across 10 arXiv macro-domains, with human-validated parent lineage (avg 8.26/10) and y∗ extraction fidelity (9.46/10).
  • Trains GIANTS-4B (Qwen3-4B base, GRPO RL) using LM-as-judge similarity as proxy reward, with strict train/test judge separation (2.5-flash for reward, 3.1-pro & Qwen3-32B for eval). Human validation: LM judge Spearman ρ=0.645 vs human ratings; SciJudge-30B prefers GIANTS-4B 68% over base.
  • GIANTS-4B consistently outperforms frontier models (gemini-3-pro, claude-opus-4-7) — 33% relative improvement on unseen-parents split; zero-shot transfers to unseen domains despite training only on Language (cs.CL); parent ablation confirms genuine two-paper synthesis (single/shuffled/no-parent collapse, reversed preserved).
Critical strengths
  1. The train/test judge separation and four-line anti-reward-hacking evidence (separate judge models, style rewriting, SciJudge-30B cross-signal, parent ablation) are thoughtfully executed and meaningfully mitigate the most natural skepticism about RL-on-LM-judge rewards.
  2. The benchmark is substantial and carefully validated — human studies confirm both y∗ extraction fidelity and parent lineage quality, and the task definition explicitly treats y∗ as one realized sample from a many-to-one distribution rather than the only "correct" insight.
  3. Zero-shot cross-domain transfer (trained only on cs.CL, evaluated across 10 domains) is a genuinely strong result that rules out simple domain-memorization explanations.
Critical weaknesses
  • The task is not insight generation — it is insight reconstruction conditioned on a specific downstream realization y∗. RL optimizes specifically to match THIS LM-extracted target, not to produce valid new scientific insights; the paper acknowledges this limitation but still frames the result as meaningful progress toward automated research synthesis, which it isn't quite.
  • Every step of the pipeline is LM-driven: parent identification, synergy extraction, paper summarization, y∗ rewriting, and similarity scoring all use commercial LMs — the human studies validate extraction fidelity but not that the RL reward correlates with insight quality beyond its own judge rubric.
  • Parent-paper identification is treated as an oracle ("we assume parent papers are provided") — in real research settings this is arguably the harder half of the problem, and GIANTS-4B's success says nothing about whether the model could select good parents itself.
Questions for authors
  • What would you estimate GIANTS-4B's performance would be on genuinely novel insight generation, i.e., when the target is not fixed to an already-published downstream paper but is instead evaluated by human experts on novelty and feasibility?
  • Why train on cs.CL alone? Does training on a more diverse corpus improve performance on cs.CL itself, and what happens if you train on a domain other than Language?
  • If the parent-paper selection were degraded (e.g., shuffled with non-foundational papers at retrieval time), how quickly does performance drop — this would bound how sensitive the synthesis step is to upstream parent quality.
2 comments

We thank Reviewer TuTy for the positive assessment of our work and for recognizing the validation of our benchmark with the anti-hacking evidence and human studies we provided as well as the transfer result we showcase beyond cs.CL. Below, we addressed the concerns you raised related to task framing, the usage of LMs, and parent paper identification/degradation.

The task is not insight generation [...] frames the result as meaningful progress toward automated research synthesis.

What would you estimate GIANTS-4B's performance would be on genuinely novel insight generation, i.e., when the target [...] is instead evaluated by human experts on novelty and feasibility?

Thank you for this question. GIANTS-4B is not designed to maximize raw novelty. Our goal is insights that are both novel and feasible, and therefore likely to have real impact once executed. Prior work shows why raw novelty is not enough: LLM-generated ideas can be judged more novel than expert ideas, yet fall behind once they are executed [1]. Our goal is literature-grounded synthesis: combining two existing works into an insight that follows from both. Novelty alone is a poor metric for this, because an insight can appear novel simply by proposing ideas that are weakly connected to, or unsupported by, the parent papers. Our qualitative analysis (Section 4 and Appendix G) shows that frontier models often do this. Their outputs are fluent but drift away from the parents or over-reach toward speculative claims the parents do not support. GIANTS-4B instead identifies a concrete, grounded connection between the two works.

This is why we anchor evaluation to realized downstream insights. Each target is a connection that researchers found worth pursuing and then validated experimentally, with some novelty over the parent papers. Training toward such targets teaches the model where the boundary between novelty and feasibility lies. This makes insight anticipation potentially useful mid-training objective that warm-starts the idea generator in automated research pipelines, rather than as a complete solution to open-ended discovery.

To the reviewer's question, our human study already evaluates insights without a fixed target. Annotators rated insights without seeing the ground-truth insight or the model identity. GIANTS-4B's insights were rated comparable to those of the base model and gemini-3-pro in engineering complexity, and significantly higher in conceptual clarity and scope, i.e., more focused and testable (Figure 6). We would therefore expect GIANTS-4B to compare favorably on feasibility in a fully open-ended expert evaluation. We would also expect it to be rated somewhat less novel than frontier models, which reflects the novelty-feasibility trade-off reported in [1] and our emphasis on grounding.

That said, novelty can also come from which parent papers are combined. Two works from distant areas that have not been connected before can lead to novel insights, and GIANTS-4B can complement such a paper-selection system by synthesizing the chosen parents into a grounded insight. We deliberately decouple parent selection from insight synthesis (Section 6), and see combining the two as a promising direction.

We agree that "insight reconstruction" describes the training setup more accurately than open-ended generation. We will tone down claims about progress toward automated research synthesis and clarify this framing in the manuscript.

Every step of the pipeline is LM-driven[…]the human studies validate extraction fidelity but not that the RL reward correlates with insight quality beyond its own judge rubric.

Thank you for this point. We agree that LMs are used throughout the pipeline, and we validate the key LM-driven steps with humans. Expert ratings confirm that the selected parents are the conceptual foundations of the downstream paper and that the extracted y∗y^* faithfully reflects each paper's stated contributions (Appendix F). For the reward itself, we have now scored the 180 human-annotated insights from Table 1 with Gemini-2.5-Flash, the judge used as the RL reward. Its scores correlate significantly with human similarity ratings (Spearman ρ=0.588\rho = 0.588, p<10−16p < 10^{-16}).

To test whether the gains extend beyond the judge's rubric, we use two signals that do not involve the similarity rubric. First, in our feasibility study (Figure 6), annotators rate insights without seeing the ground-truth insight or the model identity, and they rate GIANTS-4B's insights as having significantly higher conceptual clarity than those of the base model and gemini-3-pro. Second, SciJudge-30B, a third-party model trained on citation outcomes rather than on our rubric, prefers GIANTS-4B over the base model in 68% of comparisons.

Most importantly, our pipeline is grounded in real papers, where we extract summaries/insights from the parent and child papers to steer generation of these training and evaluation pairs.

Parent-paper identification is treated as an oracle.

We deliberately focus on insight generation given a fixed set of parent papers. Our goal is to evaluate whether models can generate novel yet tractable research insights when provided with rich contextual knowledge: specifically testing their ability to make conceptual leaps [4] and escape "idea basins" [5]. Utilizing citation graphs from published literature provides a structured warm-start objective for research agents, as human scientific progress relies on synthesizing existing work into meaningful new contributions. Furthermore, upstream parent-identification tools such as ScholarCatalyst [2] and MIR [3] can directly complement our framework. An ideal system would leverage both the ability to identify key papers as well as learning how to synthesize them in to a useful insight. We will include a detailed discussion of this design choice in the Appendix and Limitations sections to better scope our contribution.

Why train on cs.CL alone? Does training on a more diverse corpus improve performance on cs.CL itself, and what happens if you train on a domain other than Language?

Thank you for the question. Training on cs.CL alone was a design choice to measure downstream generalization: if a model trained on a single domain improves on 15 domains it has never seen, its gains cannot come from domain-specific knowledge. The results support this. GIANTS-4B improves over the base model in every domain, and its relative gains on unseen domains are as large as or larger than on cs.CL itself. Under the gemini-3-pro judge, the gain is 17.1% on cs.CL vs. 21.7% across all domains. Training on a single domain therefore does not limit transfer, and the RL objective generalizes across different domains.

We agree that training on a more diverse corpus, or on a domain other than Language, would be an informative ablation. Unfortunately, it is not feasible within the rebuttal period. Each RL run is compute-intensive and financially costly, because every rollout is scored by gemini-2.5-flash through API calls to compute the reward. We will note this as a direction for future work in Section 6.

If the parent-paper selection were degraded (e.g., shuffled with non-foundational papers at retrieval time), how quickly does performance drop?

Yes, we agree it is important to bound how sensitive the synthesis step is to upstream parent quality. Our paper actually already includes this ablation in Appendix E.2, where we degrade the parent papers given to the model at inference time. Performance drops sharply, and it drops further the more the parent information is degraded. Compared with the original setting (4.90), GIANTS-4B's similarity score falls by 39% when only one parent is provided (2.99), by 60% when the second parent is replaced with a random, non-foundational paper (1.98), and by 80% when no parents are given (1.00, the floor of the scale). Swapping the order of the two parents leaves performance unchanged (4.95).

Therefore, replacing a parent with an unrelated paper hurts more than removing it, which shows that the model attempts to synthesize whatever it is given rather than relying on one parent or on a prior over target insights.

References

[1] Idea Execution Gap (https://arxiv.org/pdf/2506.20803)

[2] ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research (https://arxiv.org/abs/2610.02202)

[3] MIR: Methodology Inspiration Retrieval for Scientific Research Problems (https://arxiv.org/abs/2506.00249)

[4] LLMs Can’t Jump (https://www.tomzahavy.com/files/llms-cant-jump.pdf)

[5] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents (https://arxiv.org/html/2610.07447v1)

Reviewer_G6wUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose the task of insight anticipation: predicting a core insight of a downstream paper given a small set of prior “parent” papers. To evaluate models in this new setup, they develop GIANTSBench, a large-scale benchmark, and a similarity-based evaluation method. The paper further introduces GIANTS-4B, a lightweight RL-trained insight generation model, which surpasses larger closed models.

Critical strengths
  • A large-scale benchmark/resource spanning diverse scientific domains.
  • Interesting and methodologically thoughtful train/test splits (temporal cutoffs, an unseen-parents subset, showing domain generalization).
  • Provides the correlation between the LLM judge and many human annotators, and reports statistical significance and error bars. Additional human studies (such as the perceived algorithmic complexity check) add to the robustness of the evaluation.
  • Showing that the trained GIANTS-4B retains its advantage over the base version under an independent quality metric (SciJudge) is valuable and strengthens the paper.
  • The paper is well-written and is easy to follow. Overall, the figures are informative and clear.
Critical weaknesses
  • In these types of problems, as you note in the limitations section, there is no single correct answer, and similarity provides a limited signal. Appendix D and the @k evaluation are important analyses in this context. An additional human study comparing the insights derived by different models (e.g., are they meaningful? feasible?) would further help you strengthen your claim that the better performance of GIANTS-4B is not an artifact of its similarity-based training, and to position it against other systems. I would also like to see a version of Figure 2 showing mean/median similarity scores.
  • Assuming only two parent papers is another drawback of your framing. I understand the context limitation, but I can think of multiple ways to overcome it, for instance, RLMs (https://arxiv.org/abs/2512.24601), agents, or even summarization. Have you considered anything similar for handling a larger number of parent papers?

Minor comments:

  • The related work section is a bit lacking, in my opinion. What about the entire line of work aimed at identifying/collecting inspiring “parent” citations (e.g., https://arxiv.org/abs/2402.16009)? I understand this is not the focus of your work, but I think briefly acknowledging it would make this section more comprehensive.
  • Radar charts are very difficult to understand. I think you should aim for another type of presentation for clarity.
Questions for authors
  • It seems that fine-tuning only on cs.CL and then testing on other domains might lower performance. Have you experimented with training and testing on the same domain?
  • I notice a large disparity between the two similarity judges (Figure 3). Have you validated both against human annotations?
  • Could you elaborate on how you provided two parent papers to SciThinker, which is designed to work with a single seed paper?
  • Have you tried computing GIANTS-4B’s SciJudge win rate against a stronger method (e.g., a larger closed model)?
4 comments

We thank Reviewer G6wU for the positive assessment of our work and for recognizing the large scale of the benchmark and the value of our evaluation with human participants. Below, we addressed some of the concerns you raised about a single correct answer and going beyond two parent papers.

There is no single correct answer, and similarity provides a limited signal.

We agree that research ideation is open-ended and that a given set of prior works can support many valid research directions. We do not claim that each pair of parent papers has a single correct insight. Instead, we study a core part of how scientists generate ideas: combining related prior work into a new contribution. Anticipating plausible downstream insights from their parent papers captures this process, and the citation graph provides a scalable, naturally occurring source of supervision for it. We therefore see insight anticipation as a way to warm-start a model’s ideation ability, for example as a mid-training objective, before moving to fully open-ended ideation, which can be grounded with execution in RSI like tasks as seen in [1].

In this framing, similarity to a realized downstream insight is an imperfect but informative proxy. It rewards insights that are grounded in the parent papers and resemble directions researchers actually pursued, and our human study shows that these scores correlate with expert ratings as seen in Table 1. The training signal is also not tied to a single target. The same parent papers recur across training examples paired with different downstream papers (37% of parent papers appear more than once). As a result, the reward reflects the range of insights that followed from those parents rather than one fixed answer.

To directly address the reliance on a single ground-truth insight, we have an evaluation in Appendix D (re-surfaced below) where because each parent-paper pair has several valid downstream insights, we view the alignment of any of the queried strategies to the target. We sampled 30 parent-paper pairs that are each shared by n ≥ 4 downstream papers and extracted the core insight of every downstream paper. Following the test-time inference setup of Figure 2, we use our LM judge to score the model-generated insight against each of the n ground-truth insights separately. Our primary metric is the maximum similarity across the n downstream papers, which measures whether the model recovers at least one direction the community actually pursued. For a fuller picture, we also report the median and mean.

ModelMax ↑Median ↑Mean ↑
Gemini 2.5 Pro4.083 ± 0.3942.958 ± 0.1413.040 ± 0.147
Gemini 3 Pro4.462 ± 0.4163.038 ± 0.1623.105 ± 0.176
DeepSeek-V2-Lite5.192 ± 0.3883.692 ± 0.2433.808 ± 0.208
Qwen3-4B5.077 ± 0.4743.346 ± 0.2673.521 ± 0.229
Qwen3-14B5.423 ± 0.4553.577 ± 0.2683.742 ± 0.249
Qwen3-32B5.385 ± 0.4843.519 ± 0.2653.663 ± 0.247
GIANTS-4B6.462 ± 0.4944.058 ± 0.3094.313 ± 0.266

GIANTS-4B achieves the highest score on all three aggregates. It outperforms its base model (Qwen3-4B), the larger Qwen3 models, DeepSeek-V2-Lite, and both Gemini Pro models. Its maximum similarity is about one point above the strongest baseline (6.46 vs. 5.42 for Qwen3-14B). This shows that the improvements from training hold when the model is credited for matching any of several valid downstream insights, not just one.

Insight anticipation is analogous to NTP (next-token prediction) for LLMs, where though we fit the model to one target prediction, we hope to model the distribution of next tokens. Therefore, through the dataset construction, we get this structure for GIANTS training and evaluation as well.

An additional human study comparing the insights derived by different models (e.g., are they meaningful? feasible?) would further help you strengthen your claim.

Thank you for this suggestion. We agree that similarity is only a partial signal for a task with many valid answers, and that a human evaluation on the insight feasibility without a reference is the right complement. The current submission already includes one such study (Section 4, Figure 6; Appendix K.2). We give its details below because they are easy to miss.

Annotators rated insights from GIANTS-4B, the base Qwen3-4B, and gemini-3-pro on two feasibility axes: (1) engineering/algorithmic complexity and (2) conceptual clarity and scope, meaning whether the insight is a focused, testable hypothesis. Annotators saw only the generated insight. They did not see the ground-truth insight, the similarity score or which model wrote it (see the interface in Figure 18). These ratings therefore do not depend on the similarity metric used in training. All three models were rated as similarly complex. GIANTS-4B was rated significantly higher on conceptual clarity than both the base model and gemini-3-pro. If RL had only taught the model to imitate the judge's preferences, we would not expect higher ratings from humans who never saw the reference.

I would also like to see a version of Figure 2 showing mean/median similarity scores.

Thank you for the suggestion. We have added a three-panel version of Figure 2 (https://imgur.com/a/GJ0J5uk). The left panel is Best@k, the original Figure 2. The middle panel is Mean@k, the average similarity score of the first k samples, averaged over examples. The right panel is Median@k, the median similarity score of the first k samples, averaged over examples. The middle and right panels show that GIANTS-4B's advantage holds for its typical sample, not only for its best one.

Going beyond two parent papers for context purposes

We agree with the reviewers framing that two going beyond two parent papers can be addressed through alternative forms of context management. We now run an experiment, where we use a stronger LLM (DeepSeek-V4.1-Flash) agentically with the pi harness (pi.dev), a common harness used by prior work [2,3] for interacting with a filesystem. Here, we supply the agent with n randomly sampled papers extracted from the related works of a target paper in a flat filesystem that the model can access and browse via the harness. We then perform inference time scaling (best@k), to understand the how the similarity evolves with more insights that are generated. We see that there is a monotonic increase in performance as more reference papers are seen, indicating that this is helpful for prediction. The best@k performance also indicates that through simple sharpening with RL, we can get to this target insight. For the final version of the manuscript and the dataset release, we will setup a filesystem variant of the evaluation and fine-tune a model with the harness.

Reference papersbest@1best@2best@4best@8
12.533.033.483.84
22.983.403.764.06
43.453.824.084.28
83.483.894.204.47
163.714.084.394.62

The related work section.

Thank you for the pointer. We will add PST-Bench and related work on identifying the sources of a paper to our related work section. PST-Bench aims to identify the prior works that most directly inspired a paper. GIANTS uses this kind of parent identification only as a data-construction step and studies the reverse direction: given the parents, can a model synthesize the insight they led to. Source-tracing methods and labels could therefore improve how GIANTSBench selects parents. If you have any other suggestions or feedback for the related works section or other parts of the manuscript, happy to accommodate those changes as well!

Radar charts are very difficult to understand.

Thank you for this feedback. We agree that radar charts with many models are a bit hard to disambiguate so we also have a set of tables as seen in Appendix C (Tables 3-5), which denote the exact numbers behind these figures, with standard errors. We are happy to swap the table with the radar plots or alternatively plot them as dot plots as seen in this anonymous url (https://imgur.com/a/OjtXvay).

Have you experimented with training and testing on the same domain?

Our setup already includes evaluation on the same-domain evaluation as training. In particular, GIANTS-4B is trained only on Language (cs.CL) papers, and our test set contains Language papers published after the training cutoff. The Language row of Tables 3 and 4 therefore measures train-and-test on the same domain, under a temporal hold-out. In this setting, GIANTS-4B improves over the base model by 17% (full test set) and 13% (Test-unseen-parents). Interestingly, these in-domain gains are smaller than the average gain across all domains (22% and 21%).

Training on one domain therefore does not narrowly produce gains that are confined to that domain. Instead, the RL objective generalizes across different domains. Additionally, Language is also one of the hardest domains for every model, so its smaller absolute gain may partly reflect difficulty. We will state this comparison explicitly in Section 5.

Disparity between the two similarity judges (Figure 3) [...] validated both against human annotations?

We have now scored the same 180 human-annotated insights used in Table 1 (including those in Figure 3) with every LM judge in the paper. Open-source and closed models of varying sizes all correlate significantly with human ratings (see the table below). The disparity in Figure 3 mainly reflects calibration rather than ranking: different judges preserve the relative ordering of models but assign different absolute scores.

JudgeSpearman ρ\rho with mean human rating
Gemini-3.1-Pro (primary)0.645
Gemini-3-Flash0.637
Gemini-2.5-Pro0.586
Gemini-2.5-Flash0.588
Qwen3-32B0.539
Qwen3-14B0.658
All correlations are significant at p<10−14p < 10^{-14} (n=180n = 180)

Could you elaborate on how you provided parent papers to SciThinker?

There are two different variants we evaluated for SciThinker. First, in “SciThinker-4B (single)”, we randomly sample one of the two identified parent papers and provide it in context, to exactly match the evaluation protocol of the paper. In “SciThinker-4B (both)”, both parent papers are received in context but we minimally modify the prompt template template with the input changed to “You have come across two new research papers with the following titles and abstracts” and two labeled blocks (Title 1/Abstract 1, Title 2/Abstract 2). The remainder of the template such as the task instruction and output format (Title/Abstract) are unchanged.

Additionally, in both variants, the abstract field holds the same parent summaries every model sees. We scored the text after Abstract: against y* with the same judge and the same best-of-k settings as Figure 2. “SciThinker-4B (both)” scores above “SciThinker-4B (single)” at every k, so SciThinker does use the second paper, but it stays below GIANTS-4B. Since SciThinker is trained for high-impact ideation rather than insight anticipation, we read this as a test of transfer, not of SciThinker on its own task. Here’s the full prompt template: https://imgur.com/a/dWTrYuB. We will add both prompts to Appendix.

References

[1] The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas (https://arxiv.org/pdf/2506.20803)

[2] Learning in a World of Notes (https://engram.com/blog/learning-in-a-world-of-notes)

[3] Harness Leaderboard (https://huggingface.co/spaces/joelniklaus/harness-leaderboard)

Thank you for the detailed and thoughtful response, and for the additional analyses and clarifications. After going through it, I’ve decided to keep my score as is.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal-Period Summary – Key Clarifications & New Results

We thank the reviewers and the AC for their thoughtful and constructive feedback. The discussion has helped us clarify the contribution of GIANTS, strengthen the validation of parent-paper selection, and improve the empirical evaluation with a larger scale human study. For the AC’s convenience, we summarize the key additions and clarifications below:

  • Methodological Contribution and Task Scope: We clarify that our contribution is the formulation of insight anticipation, the GIANTSBench benchmark, and a scalable training signal derived from the scientific citation graph. Given parent papers, the model learns to anticipate insights realized in subsequent research, using GRPO with a similarity-based reward. We position this as a potential warm-start objective for literature-grounded research ideation, often used in mid-training in LLM pipelines today. The benchmark evaluates the synthesis step; establishing benefits for end-to-end scientific discovery remains future work.

  • Validation Against Original Authors’ Reported Inspirations: To directly address concerns about the parent-papers, we evaluated our selection procedure on a third-party benchmark ScholarCatalyst, using labels provided by 184 human authors for 207 of their own papers. Of 414 selected parents, 72% were author-confirmed inspirations, 4% were explicitly rejected, and 24% were unlabeled. Among selected parents with author judgments, 95% were confirmed inspirations, compared with 44% for random selection from the same author-judged pool. These results provide direct evidence that the procedure recovers genuine inspirations, while acknowledging incomplete coverage and selection noise. We also clarify the automatic construction and exclusion rules.

  • Support for the Two-Parent Setting and Extension to Larger Reference Sets: In ScholarCatalyst, 87% of papers have at least two author-labeled inspirations, with a median of three, supporting two parents as a useful, deliberately restricted setting. Furthermore, we additionally ran a preliminary experiment on 10 target papers using an agent that browses randomly sampled reference papers in a filesystem. Increasing the reference set from one to sixteen papers raises best@1 similarity from 2.53 to 3.71 and best@8 from 3.84 to 4.62. This provides initial evidence that the task extends to broader literature contexts without requiring a preselected pair.

  • Stronger Independent Comparisons and Calibrated Performance Claims: We added SciJudge comparisons against larger open and closed models. Among pairs with consistent judgments under both presentation orders, GIANTS-4B wins 68.6% against its base model (95% CI: [67.4, 69.9]), approximately 55% against supervised fine-tuning variants, and 54.5% against Qwen3-14B. Frontier closed models remain preferred under SciJudge’s citation-impact criterion. These results sharpen our claim: training improves the 4B model on both insight anticipation and an independent evaluation signal, while benchmark leadership does not establish general superiority in research-idea quality.

  • Human Validation and Robustness Across Judges: We evaluated all six similarity judges on the same 180 human-annotated insights. Their scores correlate significantly with human ratings, with Spearman correlations ranging from 0.539 to 0.658; the RL reward judge achieves 0.588. Beyond similarity, our existing blinded human study evaluates insights without revealing the reference or model identity. Annotators rate GIANTS-4B’s insights as significantly clearer and better scoped than those of the base model and Gemini-3-Pro, at comparable engineering complexity. These findings provide complementary evidence beyond the training rubric.

We are confident that these additions and clarifications have addressed the major points raised during the review process. The constructive dialogue has substantially improved the paper, and we are grateful for the reviewers' engagement. We respectfully ask the AC to consider these clarifications and additional results in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.