- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces PRESCIENCE to evaluate scientific forecasting. The dataset covers AI and quantum physics, with about 1.06M papers including historical literature. It defines seven tasks covering paper content, collaborations, citations, and topic trends. The authors compare several baselines and introduce LACER to assess generated contributions. They also simulate a year of publications and find lower diversity and novelty than in real papers.
Critical strengths
Future combination prediction and the year-long rollout are somewhat interesting, examining research connections and repetition in generated papers. (But I still think their actual value and scope of applicability is unclear.)
Critical weaknesses
-
The overall motivation is weak. I think the paper does not clearly explain which research decisions would benefit from better scores.
-
Collaborator prediction primarily captures existing partnerships and rarely identifies first-time collaborators. But what does this mean? Inaccurate predictions may not necessarily indicate a poor model performance.
-
Prior work selection uses only the author team, without a research question or abstract. Its use case is unclear, and it may mainly capture past citation habits.
-
Citation prediction adds limited value, and the paper does not clearly explain what it reveals beyond existing works.
-
Trend forecasting evaluates only simple baselines, leaving unclear which model designs the dataset can support.
-
Human validation of LACER covers only ten target papers, with annotators drawn from the authors, limiting its scope and independence.
Questions for authors
-
Which concrete research decisions would benefit from more accurate predictions on the tasks presented in this paper?
-
Several tasks have substantial prior work. For example, citation count prediction has been widely studied, and contribution generation has closely related prior work. What new insights are gained by evaluating these tasks on a shared dataset?
5 comments
Response to Reviewer TwMp (2/2)
W6 continued)
- Human evaluation (Appendix C.3, Table 12): five experts each ranked 10 candidate abstracts per target paper. LACER's agreement with experts (Kendall 0.57, 95% CI 0.46–0.66) is at least as high as inter-annotator agreement (0.53, CI 0.38–0.63): slightly higher when bootstrapping over target papers, and not significantly different when bootstrapping over individual rankings. BERTScore and ASPIRE fall significantly below it.
- Control baselines (Table 2): scores follow the expected order: a paper from the same arXiv category 1.26, a random influential reference 4.23, GPT-5 family models 5.41–5.63, and a paraphrase of the target abstract 10.00.
- Judge robustness (Appendix C.2.1): on an earlier version of our dataset (papers from Oct 2024 to Sep 2025), Claude Opus 4.5, GLM 5.2 and Gemini 3.5 Flash judges reproduce the GPT-5 judge's average score per model (Pearson 0.985–0.996 across models) and model ranking.
Q1) Please see W1.
Q2) As all tasks share one temporally aligned dataset, we can run holistic analyses across them. Each paper-anchored task predicts one field of a paper record, and its output is the next task's input: collaborator prediction turns a seed author into a team, prior work selection turns the team into influential references, contribution generation turns those into a title and abstract, and citation count prediction predicts citations from the completed record. Since every task uses only earlier literature, generated papers can be added to the literature for later predictions. In Section 4.5, we generate 12 months of synthetic papers this way and compare them with real papers from the same period. ResearchTown [12] and AgentRxiv [13] also generate synthetic research but do not compare it with real papers, and we know of no prior work that does. We find:
- Synthetic papers are less diverse (5.35 ± 0.07 vs. 6.15 ± 0.10) and less novel relative to the evolving literature (3.79 ± 0.06 vs. 4.29 ± 0.10) than real ones ( for both; 95% CIs; Section 4.5).
- Most of the novelty drop disappears against the pre-rollout literature, suggesting generated papers become similar to earlier papers.
- Predicted collaborators and references are more diverse than real ones (1.72× and 1.60×; Appendix E.6.1). Even with the true references and authors as input, generated papers are less diverse and less novel than their real counterparts ( for both; Appendix E.6.1, Table 23), so the gap comes mainly from contribution generation. Prompting for novelty or diversity, or retrieving similar papers, does not close it (see Reviewer wzTF, Q3).
- Precision and recall against real papers are about one LACER point below a real-vs-real reference, and the best-covered real papers tend to be more highly cited, suggesting a bias toward mainstream work (Appendix E.6.3).
The shared dataset also shows which inputs help: for contribution generation, author histories help most (LACER 5.61 to 5.85, ; Table 2), while oracle citation counts do not (5.57). Since PreScience can be renewed, such analyses remain possible for future models.
[1] He-Yueya et al. GIANTS: Generative Insight Anticipation from Scientific Literature. arXiv 2026.
[2] Wang et al. Learning to Predict Future-Aligned Research Proposals with Language Models. arXiv 2026.
[3] Liben-Nowell and Kleinberg. The Link Prediction Problem for Social Networks. CIKM 2003.
[4] Zhang et al. Scholarly Recommendation Systems: A Literature Survey. Knowledge and Information Systems, 2023.
[5] Wang, Song and Barabási. Quantifying Long-Term Scientific Impact. Science, 2013.
[6] Hirako et al. Realistic Citation Count Prediction Task for Newly Published Papers. Findings of EACL 2023.
[7] Shi and Evans. Surprising Combinations of Research Contents and Contexts Are Related to Impact and Emerge with Scientific Outsiders from Distant Disciplines. Nature Communications, 2023.
[8] Chen et al. Structuring Scientific Innovation: A Framework for Modeling and Discovering Impactful Knowledge Combinations. arXiv 2025.
[9] Salatino, Mannocci and Osborne. Detection, Analysis, and Prediction of Research Topics with Scientific Knowledge Graphs. Predicting the Dynamics of Research Impact, Springer, 2021.
[10] Ofer, Kaufman and Linial. What's Next? Forecasting Scientific Research Trends. Heliyon, 2023.
[11] Gu and Krenn. Forecasting High-Impact Research Topics via Machine Learning on Evolving Knowledge Graphs. Machine Learning: Science and Technology, 2025.
[12] Yu et al. ResearchTown: Simulator of Human Research Community. ICML 2025.
[13] Schmidgall and Moor. AgentRxiv: Towards Collaborative Autonomous Research. arXiv 2025.
[14] Uzzi et al. Atypical Combinations and Scientific Impact. Science, 2013.
[15] Ansari et al. Chronos: Learning the Language of Time Series. TMLR 2024.
[16] Ansari et al. Chronos-2: From Univariate to Universal Forecasting. arXiv 2025.
I'm still unsure why these forecasting tasks are necessary for scientific research decision-making.
For example, the author mentions that using a link prediction protocol is reasonable for tasks like collaborator prediction, but I don't understand whether it can be used for collaborator recommendation.
The model only knows the researcher's identity and the prediction date, yet it must predict the co-authors of that researcher's next paper.
In other words, the model doesn't know the research topic, research question, or collaboration needs of the next paper.
Therefore, this task likely relies primarily on the researcher's past collaboration record, rather than determining which researchers are academically suitable for collaboration.
For example, suppose a professor will co-author a paper with Professor A in the future.
Model 1 predicts A, Model 2 predicts B.
According to the author's assessment, Model 1 is better.
However, suppose Professor B actually has more complementary expertise, but the two didn't have the opportunity to meet, or were limited by factors such as funding, time, or geographical location, and ultimately didn't collaborate.
Then, Model 2's prediction might be more valuable for collaborator recommendation.
However, it's important to distinguish that if the task were simply to predict "who will actually collaborate," Model 1 would indeed be more accurate. The problem isn't that the ground truth is ineffective for this prediction task, but rather that this prediction task differs from the collaborator's recommended goal.
Model prediction failures can have at least two causes:
- First, the model hasn't fully utilized observable research information.
- Second, future collaborations inherently depend heavily on information not present in the data.
Response to Reviewer TwMp (1/2)
Thank you for your review. We have incorporated your comments into our revised draft (changes shown in blue) and believe they improve the paper. We address each Weakness (W) and Question (Q) below.
W1) Four of our tasks (contribution generation [1, 2], collaborator prediction [3, 4], citation prediction [5, 6] and topic trends [9–11]) are long-studied problems, showing their importance to the community. The other three are new formulations that our shared schema makes possible. We do not evaluate these applications end-to-end, but PreScience measures the predictive capabilities they rely on.
Since the dataset can be renewed past each new model's training cutoff, all of these evaluations stay free of contamination.
W2) We follow the standard link prediction protocol [3], evaluating against the co-authors of a real future paper. We agree that some plausible collaborators may be missing from the ground truth. This makes scores conservative, but every method is evaluated the same way on many datapoints, so comparisons between methods are meaningful. Appendix E.2.3 shows that current methods predict repeat collaborations well but rarely predict new ones (i.e., established authors who have not co-authored with the seed before). We believe this makes PreScience more useful as a benchmark, since it highlights gaps in current systems. We have also added classic link prediction baselines (see Reviewer ekLW, W2).
W3) Given a team of collaborators, this task surfaces prior work likely to be influential for their next paper (Section 3), e.g., for a new group or a lab planning its next project before a research question is settled. The abstract is withheld because it describes the paper that builds on those references and would leak the answer; withholding it also lets the task be chained in the rollout (Section 4.5), where references are chosen before any abstract exists. As the reviewer notes, past citation habits are a strong signal, so Frequency performs well, while all methods score near zero on references the team has never cited (Appendix E.3.3). Autoresearch improves nDCG from 0.09 (Frequency) to 0.13, so going beyond these habits remains difficult. We think this makes PreScience more interesting as a benchmark.
W4) Our shared dataset shows what drives citation prediction: adding team and reference text improves 6-month Spearman from 0.37 to 0.42 on AI (Table 4). Zero-shot GPT-5, whose training cutoff precedes our test papers, beats bibliometric regressors without any training labels (Spearman 0.39 vs. 0.33 on AI; see Reviewer wzTF, W2).
W5) Following your feedback, we added GPT-5 (training cutoff before the test period) prompted with each topic's name and monthly history, two time-series foundation models (Chronos-Bolt [15] and Chronos-2 [16]), and Autoresearch (a 12-hour run tuned on a training-period dev split). We also replaced our taxonomy with a fully automatic one for both domains (Appendix B.3). Unary Δshare R² / Pearson:
None of the added methods beat Linear Extrapolation on AI. Autoresearch overfits its dev split (R² 0.68 on dev, −0.62 on test). Since the dataset also includes author, citation and text information for every paper, it can support models that combine these signals. Topic-pair results are in our response to Reviewer ekLW, W1. We have added these to Tables 5 and 29.
W6) We believe three analyses together suffice to support our use of LACER: