acceptodds
Under review as a conference paper at ICLR 2027

PreScience: A Dataset and Benchmark for Scientific Forecasting

Abstract

Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a renewable dataset and benchmark for scientific forecasting built around 147K recent research papers from multiple domains, together with companion papers covering author publication histories and citation links, yielding 1.06M papers in total. The resulting paper records include titles, abstracts, disambiguated author identities, influential references, topic labels, citation trajectories, and metadata snapshotted to respect temporal cutoffs. We instantiate seven exemplar tasks: five paper-anchored tasks—contribution generation, collaborator prediction, prior work selection, citation count prediction, and future combination prediction—and two aggregate topic trend forecasting variants. We develop baselines ranging from simple heuristics and embedding methods to frontier language models and agentic systems, and introduce LACER, an LLM-based metric for evaluating similarity of generated contribution descriptions that agrees better with human judgments than existing metrics. Finally, we compose task models into a 12-month synthetic corpus whose papers are systematically less diverse and less novel than human-authored research from the same period.

Primary area:
datasets and benchmarks
Keywords:
science, forecasting, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TwMpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PRESCIENCE to evaluate scientific forecasting. The dataset covers AI and quantum physics, with about 1.06M papers including historical literature. It defines seven tasks covering paper content, collaborations, citations, and topic trends. The authors compare several baselines and introduce LACER to assess generated contributions. They also simulate a year of publications and find lower diversity and novelty than in real papers.

Critical strengths

Future combination prediction and the year-long rollout are somewhat interesting, examining research connections and repetition in generated papers. (But I still think their actual value and scope of applicability is unclear.)

Critical weaknesses
  • The overall motivation is weak. I think the paper does not clearly explain which research decisions would benefit from better scores.

  • Collaborator prediction primarily captures existing partnerships and rarely identifies first-time collaborators. But what does this mean? Inaccurate predictions may not necessarily indicate a poor model performance.

  • Prior work selection uses only the author team, without a research question or abstract. Its use case is unclear, and it may mainly capture past citation habits.

  • Citation prediction adds limited value, and the paper does not clearly explain what it reveals beyond existing works.

  • Trend forecasting evaluates only simple baselines, leaving unclear which model designs the dataset can support.

  • Human validation of LACER covers only ten target papers, with annotators drawn from the authors, limiting its scope and independence.

Questions for authors
  • Which concrete research decisions would benefit from more accurate predictions on the tasks presented in this paper?

  • Several tasks have substantial prior work. For example, citation count prediction has been widely studied, and contribution generation has closely related prior work. What new insights are gained by evaluating these tasks on a shared dataset?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TwMp (1/2)

Thank you for your review. We have incorporated your comments into our revised draft (changes shown in blue) and believe they improve the paper. We address each Weakness (W) and Question (Q) below.

W1) Four of our tasks (contribution generation [1, 2], collaborator prediction [3, 4], citation prediction [5, 6] and topic trends [9–11]) are long-studied problems, showing their importance to the community. The other three are new formulations that our shared schema makes possible. We do not evaluate these applications end-to-end, but PreScience measures the predictive capabilities they rely on.

TaskUsePrior work
Contribution generationEvaluating scientific ideation, hypothesis selection and automated discovery[1, 2]
Collaborator predictionCollaborator recommendation, author discovery, and how communities form around emerging problems[3, 4]
Prior work selectionHelping teams find prior work and directions to build onNew task
Citation count predictionAnticipating which papers will get early scholarly attention[5, 6]
Future combinationIdentifying which lines of work can productively be combined, a long-standing science-of-science question [14]New formulation; related: [7, 8]
Topic trendsHelping scientists and policymakers track which areas gain or lose attention[9–11] (pairs: new)
Corpus rolloutsStudying how AI-generated research could change the literature (e.g., homogenization)[12, 13]

Since the dataset can be renewed past each new model's training cutoff, all of these evaluations stay free of contamination.

W2) We follow the standard link prediction protocol [3], evaluating against the co-authors of a real future paper. We agree that some plausible collaborators may be missing from the ground truth. This makes scores conservative, but every method is evaluated the same way on many datapoints, so comparisons between methods are meaningful. Appendix E.2.3 shows that current methods predict repeat collaborations well but rarely predict new ones (i.e., established authors who have not co-authored with the seed before). We believe this makes PreScience more useful as a benchmark, since it highlights gaps in current systems. We have also added classic link prediction baselines (see Reviewer ekLW, W2).

W3) Given a team of collaborators, this task surfaces prior work likely to be influential for their next paper (Section 3), e.g., for a new group or a lab planning its next project before a research question is settled. The abstract is withheld because it describes the paper that builds on those references and would leak the answer; withholding it also lets the task be chained in the rollout (Section 4.5), where references are chosen before any abstract exists. As the reviewer notes, past citation habits are a strong signal, so Frequency performs well, while all methods score near zero on references the team has never cited (Appendix E.3.3). Autoresearch improves nDCG from 0.09 (Frequency) to 0.13, so going beyond these habits remains difficult. We think this makes PreScience more interesting as a benchmark.

W4) Our shared dataset shows what drives citation prediction: adding team and reference text improves 6-month Spearman from 0.37 to 0.42 on AI (Table 4). Zero-shot GPT-5, whose training cutoff precedes our test papers, beats bibliometric regressors without any training labels (Spearman 0.39 vs. 0.33 on AI; see Reviewer wzTF, W2).

W5) Following your feedback, we added GPT-5 (training cutoff before the test period) prompted with each topic's name and monthly history, two time-series foundation models (Chronos-Bolt [15] and Chronos-2 [16]), and Autoresearch (a 12-hour run tuned on a training-period dev split). We also replaced our taxonomy with a fully automatic one for both domains (Appendix B.3). Unary Δshare R² / Pearson:

BaselineAIQuantum
Linear Extrapolation0.41 / 0.71−2.43 / 0.13
Citation Momentum−4.24 / 0.44−11.78 / 0.28
Chronos-Bolt0.05 / 0.23−1.04 / 0.01
Chronos-20.00 / 0.13−0.75 / 0.04
GPT-5−0.05 / 0.30−0.95 / 0.10
Autoresearch−0.62 / 0.09−3.53 / −0.08

None of the added methods beat Linear Extrapolation on AI. Autoresearch overfits its dev split (R² 0.68 on dev, −0.62 on test). Since the dataset also includes author, citation and text information for every paper, it can support models that combine these signals. Topic-pair results are in our response to Reviewer ekLW, W1. We have added these to Tables 5 and 29.

W6) We believe three analyses together suffice to support our use of LACER:

AuthorsOpenReview ↗

Response to Reviewer TwMp (2/2)

W6 continued)

  1. Human evaluation (Appendix C.3, Table 12): five experts each ranked 10 candidate abstracts per target paper. LACER's agreement with experts (Kendall τb\tau_b 0.57, 95% CI 0.46–0.66) is at least as high as inter-annotator agreement (0.53, CI 0.38–0.63): slightly higher when bootstrapping over target papers, and not significantly different when bootstrapping over individual rankings. BERTScore and ASPIRE fall significantly below it.
  2. Control baselines (Table 2): scores follow the expected order: a paper from the same arXiv category 1.26, a random influential reference 4.23, GPT-5 family models 5.41–5.63, and a paraphrase of the target abstract 10.00.
  3. Judge robustness (Appendix C.2.1): on an earlier version of our dataset (papers from Oct 2024 to Sep 2025), Claude Opus 4.5, GLM 5.2 and Gemini 3.5 Flash judges reproduce the GPT-5 judge's average score per model (Pearson 0.985–0.996 across models) and model ranking.

Q1) Please see W1.

Q2) As all tasks share one temporally aligned dataset, we can run holistic analyses across them. Each paper-anchored task predicts one field of a paper record, and its output is the next task's input: collaborator prediction turns a seed author into a team, prior work selection turns the team into influential references, contribution generation turns those into a title and abstract, and citation count prediction predicts citations from the completed record. Since every task uses only earlier literature, generated papers can be added to the literature for later predictions. In Section 4.5, we generate 12 months of synthetic papers this way and compare them with real papers from the same period. ResearchTown [12] and AgentRxiv [13] also generate synthetic research but do not compare it with real papers, and we know of no prior work that does. We find:

  1. Synthetic papers are less diverse (5.35 ± 0.07 vs. 6.15 ± 0.10) and less novel relative to the evolving literature (3.79 ± 0.06 vs. 4.29 ± 0.10) than real ones (p<10−15p < 10^{-15} for both; 95% CIs; Section 4.5).
  2. Most of the novelty drop disappears against the pre-rollout literature, suggesting generated papers become similar to earlier papers.
  3. Predicted collaborators and references are more diverse than real ones (1.72× and 1.60×; Appendix E.6.1). Even with the true references and authors as input, generated papers are less diverse and less novel than their real counterparts (p<10−4p < 10^{-4} for both; Appendix E.6.1, Table 23), so the gap comes mainly from contribution generation. Prompting for novelty or diversity, or retrieving similar papers, does not close it (see Reviewer wzTF, Q3).
  4. Precision and recall against real papers are about one LACER point below a real-vs-real reference, and the best-covered real papers tend to be more highly cited, suggesting a bias toward mainstream work (Appendix E.6.3).

The shared dataset also shows which inputs help: for contribution generation, author histories help most (LACER 5.61 to 5.85, p<0.001p < 0.001; Table 2), while oracle citation counts do not (5.57). Since PreScience can be renewed, such analyses remain possible for future models.

[1] He-Yueya et al. GIANTS: Generative Insight Anticipation from Scientific Literature. arXiv 2026.

[2] Wang et al. Learning to Predict Future-Aligned Research Proposals with Language Models. arXiv 2026.

[3] Liben-Nowell and Kleinberg. The Link Prediction Problem for Social Networks. CIKM 2003.

[4] Zhang et al. Scholarly Recommendation Systems: A Literature Survey. Knowledge and Information Systems, 2023.

[5] Wang, Song and Barabási. Quantifying Long-Term Scientific Impact. Science, 2013.

[6] Hirako et al. Realistic Citation Count Prediction Task for Newly Published Papers. Findings of EACL 2023.

[7] Shi and Evans. Surprising Combinations of Research Contents and Contexts Are Related to Impact and Emerge with Scientific Outsiders from Distant Disciplines. Nature Communications, 2023.

[8] Chen et al. Structuring Scientific Innovation: A Framework for Modeling and Discovering Impactful Knowledge Combinations. arXiv 2025.

[9] Salatino, Mannocci and Osborne. Detection, Analysis, and Prediction of Research Topics with Scientific Knowledge Graphs. Predicting the Dynamics of Research Impact, Springer, 2021.

[10] Ofer, Kaufman and Linial. What's Next? Forecasting Scientific Research Trends. Heliyon, 2023.

[11] Gu and Krenn. Forecasting High-Impact Research Topics via Machine Learning on Evolving Knowledge Graphs. Machine Learning: Science and Technology, 2025.

[12] Yu et al. ResearchTown: Simulator of Human Research Community. ICML 2025.

[13] Schmidgall and Moor. AgentRxiv: Towards Collaborative Autonomous Research. arXiv 2025.

[14] Uzzi et al. Atypical Combinations and Scientific Impact. Science, 2013.

[15] Ansari et al. Chronos: Learning the Language of Time Series. TMLR 2024.

[16] Ansari et al. Chronos-2: From Univariate to Universal Forecasting. arXiv 2025.

I'm still unsure why these forecasting tasks are necessary for scientific research decision-making.

For example, the author mentions that using a link prediction protocol is reasonable for tasks like collaborator prediction, but I don't understand whether it can be used for collaborator recommendation.

The model only knows the researcher's identity and the prediction date, yet it must predict the co-authors of that researcher's next paper.

In other words, the model doesn't know the research topic, research question, or collaboration needs of the next paper.

Therefore, this task likely relies primarily on the researcher's past collaboration record, rather than determining which researchers are academically suitable for collaboration.

For example, suppose a professor will co-author a paper with Professor A in the future.

Model 1 predicts A, Model 2 predicts B.

According to the author's assessment, Model 1 is better.

However, suppose Professor B actually has more complementary expertise, but the two didn't have the opportunity to meet, or were limited by factors such as funding, time, or geographical location, and ultimately didn't collaborate.

Then, Model 2's prediction might be more valuable for collaborator recommendation.

However, it's important to distinguish that if the task were simply to predict "who will actually collaborate," Model 1 would indeed be more accurate. The problem isn't that the ground truth is ineffective for this prediction task, but rather that this prediction task differs from the collaborator's recommended goal.

Model prediction failures can have at least two causes:

  • First, the model hasn't fully utilized observable research information.
  • Second, future collaborations inherently depend heavily on information not present in the data.
Reviewer_wzTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PreScience, an updateable scientific forecasting benchmark built by integrating research papers, authors’ publication histories, and citation data in AI and quantum physics. It evaluates various baselines on seven tasks: contribution generation, collaborator prediction, prior work selection, citation count prediction, future combination prediction, and two forms of topic trend forecasting. It further introduces LACER to assess the similarity between generated and actual contributions and validates the metric using human judgments. By composing prediction models into a corpus-generation pipeline, the authors find that the resulting synthetic literature is less diverse and less novel than real research papers.

Critical strengths
  • By providing an integrated corpus of scientific papers and defining multiple forecasting tasks on a shared data foundation, this work offers a potentially valuable resource for advancing research in scientific forecasting.
Critical weaknesses
  • The paper does not sufficiently establish a substantial contribution beyond assembling existing resources and defining prediction tasks over them. The added value of the dataset beyond the integration of existing resources is not sufficiently demonstrated, and the individual tasks exhibit either limited novelty or insufficient validation of their formulations. Consequently, the overall contribution remains limited despite the potential utility of the integrated resource.
  • While the breadth of the baseline evaluation is valuable, the methods used for the individual tasks are not sufficiently developed to support strong conclusions. In particular, the evaluation of established tasks such as citation count prediction omits comparisons with relevant methods from prior work. More importantly, the central finding described as “observing systematic degradations in diversity and novelty relative to human-authored research from the same period” is not adequately disentangled from the limitations of the adopted methods. The experiments do not establish whether these degradations result from the chosen methods or reflect a more fundamental difficulty inherent in the task formulation.
Questions for authors
  • My understanding is that contribution generation uses only the titles and abstracts of the target paper’s influential references. Does this restriction mainly reflect constraints on full-text availability? Since researchers generally consult the full texts of relevant studies when developing research ideas, could you explain why this restricted input is appropriate for the intended evaluation? Also, the cited papers may have been included in the LLMs’ training data, allowing the models to draw on their full-text contents despite receiving only titles and abstracts as explicit input. How does this possibility affect the interpretation of the results?
  • For citation count prediction, even if the test papers themselves are absent from the training data, the citation counts used as labels for training papers may include citations that were not yet observable when the test papers were published. Prior work has shown that this form of temporal leakage can affect performance (Hirako et al., 2023, Findings of EACL). Have you taken measures to prevent such leakage?
  • Have you evaluated mechanisms that explicitly promote diversity and novelty during corpus generation, such as diversity-aware reference selection or novelty-oriented generation prompts? Would such mechanisms reduce the observed gap relative to human-authored research? If these settings have not been tested, could you clarify the scope of the conclusions supported by the current pipeline?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wzTF (1/2)

Thank you for your review. We have incorporated your comments into our revised draft (changes shown in blue) and believe they improve the paper. We address each Weakness (W) and Question (Q) below.

W1) A carefully integrated dataset at this scale (>140K target papers, >1M papers in total, >200K disambiguated authors) and quality is a valuable contribution in itself, as many widely used benchmarks show. However, PreScience also goes further, in five ways:

  1. Simulating the trajectory of science: To the best of our knowledge, our work is the first to simulate the trajectory of science at the scale of tens of thousands of papers and compare it against the real literature. Each paper-anchored task predicts one field of a shared paper record, and each task's output is the next task's input. Since every task uses only literature dated before the paper, generated papers can be added to the literature to condition later predictions. Our task definitions allow for composing models this way, and doing so (Section 4.5) shows that AI-generated papers are less novel and less diverse than real papers from the same period. We are optimistic that better methods for this new task will be introduced in future work, and facilitating that work is a primary goal of our benchmark.
  2. New tasks: Three of our seven tasks are new formulations that our shared schema makes possible: team-conditioned prior work selection, predicting which pairs of papers will be built on together, and topic-pair trends.
  3. A validated metric for scientific contributions: Measuring how close a generated contribution is to what was actually published is itself an open problem, and standard text-similarity metrics do it poorly. LACER agrees with expert rankings as well as experts agree with each other (Kendall τb\tau_b 0.57 vs. 0.53), well above ROUGE-L, BERTScore and ASPIRE (0.27 / 0.40 / 0.35). It also orders control baselines as expected and is robust to the choice of judge LLM (Appendix C.2.1). This makes contribution generation, and the corpus-level novelty and diversity of generated science, measurable at scale.
  4. Fully automatic and renewable: Building the dataset is a nontrivial task that involves temporally aligning papers, authors, references and citations, and disambiguating authors at scale. We have constructed a pipeline that not only achieves this but is also fully automatic and renewable by construction (see Reviewer ekLW, W1), which is necessary to keep evaluating frontier systems.
  5. Extensive evaluation: We evaluate heuristic, embedding, LLM, agentic and Autoresearch baselines on all seven tasks in two domains. Our analyses show where current systems fail (e.g., in predicting new collaborators and unfamiliar references, beating simple extrapolation on topic trends, and generating papers as novel and diverse as real ones).

W2) Thank you for this suggestion. Some of our baselines already follow prior work (e.g., Bibliometrics follows [1]). We have added three more from the literature: negative binomial regression [2], zero-shot GPT-5 [3], and NAIP [4], a released LLaMA-3-8B model fine-tuned for impact prediction. If there are specific methods you had in mind, we would be glad to add them too. Following [5], we also remove temporal leakage from the training labels (see Q2). Leak-free 6-month Spearman:

MethodAIQuantum
Bibliometrics, XGBoost [1]0.330.17
Bibliometrics, negative binomial [2]0.250.15
Zero-shot GPT-5 [3]0.390.24
NAIP, released model [4]0.170.14
Best trained baseline (Q2)0.430.29

We have added these results to Table 4. We have also added link prediction baselines for collaborator prediction and prior work selection, and LLM, Chronos and Autoresearch baselines for topic trend forecasting (see Reviewer ekLW, W1 and W2).

AuthorsOpenReview ↗

Response to Reviewer wzTF (2/2)

W2 continued) Regarding the rollouts: thank you for the comment. In response, we have extended our experiments (Appendix E.6.2), and we have re-scoped our claims (Sections 4.5 and 6) to avoid implying more generality than our experiments support. To the best of our knowledge, our work is the first to simulate the trajectory of science at the scale of tens of thousands of papers, and a complete exploration of methods for this new task is outside the scope of our paper. We are optimistic that better methods will be introduced in future work, and facilitating that work is a primary goal of our benchmark. Following your feedback, we evaluated three additional pipelines that explicitly encourage novelty and diversity, including one that retrieves similar existing papers (see Q3). Across all four pipelines and six rollouts, synthetic papers are consistently less diverse than real papers and less novel relative to the evolving literature. Prompting and retrieval improve novelty but not diversity, suggesting these gaps reflect a robust limitation of current LLM-generated research rather than a quirk of one pipeline.

Q1) Though titles and abstracts do represent a limited view into a paper, they contain the core motivation and contributions, and are carefully written to be representative of a paper’s contents. Offering full text representations of papers in our dataset would cause the dataset size to increase dramatically (a typical full-text is 39x longer than the corresponding abstract) and also dramatically increase the cost of evaluations conducted over this data (mean 4K -> 158K characters per target; the longest reach 670K, near context limits). To measure the effect of this choice, we ran an ablation that provided the influential references’ full texts as input in the contribution generation task rather than abstracts, and observed no change in mean LACER scores.

InputLACER
Infl. refs, titles + abstracts (paper)5.61 ± 0.20
Infl. refs, + full text5.61 ± 0.20

We have added this result to Table 2.

Regarding memorization, our target papers postdate the training cutoffs of six of the nine models we evaluate, including GPT-5 mini, which we use for the rollouts. For the other three, LACER scores just before and just after the cutoff do not differ significantly (Appendix E.1.1).

Q2) Thank you for pointing this out. This leakage was indeed present. We updated our task formulation to predict 6-month citation counts, and use only data from the first 6 months of the train period to train models. This resolves the leakage. Leak-free 6-month Spearman:

BaselineAIQuantum
Target Text0.370.25
Bibliometrics0.330.17
Target + Context0.420.27
T + C + Bibliometrics0.430.29
Autoresearch0.430.27

We find that the updated citation prediction results are largely qualitatively similar, and we have updated Table 4 accordingly, along with the new baselines from W2.

Q3) Following your suggestion, we tested three variants of the generation prompt. We targeted contribution generation since our analysis (Appendix E.6.1, Fig. 13) shows that predicted collaborators and references are already more diverse than their real counterparts. The variants add, cumulatively: (a) instructions explicitly asking for novelty, (b) instructions explicitly asking for diversity, and (c) for each influential reference, the 3 most similar pre-dated papers in the corpus being generated, with instructions to differ meaningfully from them.

CorpusDiversityNovelty vs. H<t
Natural6.154.29
Synthetic, paper (3 seeds / seed 42)5.35 / 5.433.79 / 3.85
+ novelty (a)5.444.13
+ novelty + diversity (a, b)5.334.05
+ novelty + diversity + retrieval (a, b, c)5.294.01

Asking for novelty closes up to two-thirds of the novelty gap but leaves a statistically significant gap (p≤0.035p \leq 0.035), and no variant changes diversity, which stays 0.7–0.9 below real papers in every run (p<10−16p < 10^{-16}). Generations move away from prior literature but not from each other. We have added these results to Appendix E.6.2 (Table 24).

[1] Yan et al. Citation Count Prediction: Learning to Estimate Future Citations for Literature. CIKM 2011.

[2] Didegah and Thelwall. Determinants of Research Citation Impact in Nanoscience and Nanotechnology. JASIST, 2013.

[3] de Winter. Can ChatGPT Be Used to Predict Citation Counts, Readership, and Social Media Interaction? Scientometrics, 2024.

[4] Zhao et al. From Words to Worth: Newborn Article Impact Prediction with LLM. AAAI 2025.

[5] Hirako et al. Realistic Citation Count Prediction Task for Newly Published Papers. Findings of EACL 2023.

Thank you for your detailed responses to the weaknesses and questions I raised. In particular, I found the additional experimental results addressing Q1, Q2, and Q3 very useful, and they positively influenced my assessment of the paper. However, my concerns about the validity of the newly added tasks remain unresolved, so I will maintain my original score.

AuthorsOpenReview ↗

Followup response to Reviewer wzTF

We are pleased to hear that our answers to your questions were satisfactory. We appear to have misunderstood—we thought the validation concerns you raised could be addressed by answering your questions Q1-Q3, but it appears your concerns were separate from those questions. Can you please help us understand your validation concerns and what steps we could take to address them?

Reviewer_ekLWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes PreScience, a renewable dataset and benchmark for scientific forecasting, with 147K arXiv target papers in AI and Quantum Physics. Specifically, the authors define 7 forecasting tasks on individual papers and topic trends, and chain some of them to generate synthetic papers for 12 months. They also propose LACER, an LLM-judge metric for contribution generation, which agrees with human rankings about as well as annotators agree with each other. From the results on heuristic, embedding, LLM, and agentic baselines, the authors find that (1) agentic methods perform the best on the ranking tasks; (2) the generated papers are less diverse and less novel than real papers.

Critical strengths
  1. The dataset is very rich, which links 147K target papers to their authors, references, and citation trajectories.

  2. The authors conduct extensive experiments on 7 tasks, with a lot of baselines from simple heuristics to LLMs and agents.

Critical weaknesses
  1. The authors stress a lot that PreScience is renewable, and even claim that “a forecasting benchmark must be renewable”. However, how to renew it is only mentioned in one sentence of the conclusion and a footnote, where the authors say the scripts need only a date range and arXiv categories. It seems that some steps still need manual work. For example, the 202-topic AI taxonomy was refined manually, and there is no taxonomy for Quantum Physics at all, so the topic trends are not evaluated there. So what will the users do to make sure that the dataset can be renewed? If this needs a lot of manual work, then the contributions of this paper may be impacted.

  2. The authors formulate collaborator prediction as a link-prediction problem. However, classic link-prediction baselines, such as common neighbors, Adamic-Adar, and Katz, are missing. As far as I understand, the best Autoresearch pipeline on this task mainly counts the seed author’s frequent co-authors in recent papers, and expands them through 2-hop and 3-hop co-author links. This looks very similar to a truncated Katz index. These simple baselines should be added. Otherwise, it is hard to tell whether the claim in Section 4.2 that “agentic methods perform best across all three ranking tasks” holds for collaborator prediction.

Questions for authors
  1. Which steps in the released scripts run automatically, and which need manual work or paid API access? Also, how much does it cost to renew the dataset once?

  2. From my understanding, the historical record before each prediction time is built from a corpus that only contains papers “reachable from at least one target paper”, and the targets include test papers posted up to July 2026. Is this correct? If so, which papers and authors show up in the history also depends on papers posted later. For example, the true collaborators in collaborator prediction are co-authors of the target paper, so their prior papers are always included, unlike many other candidates. This may favor the co-author expansion in Autoresearch. How many candidates and true collaborators can only be reached through the target paper or later targets?

  3. The topic trend tasks are the only ones without an LLM or agentic baseline. I’m curious to see whether an LLM with a cutoff before the test period can tell which topics will rise.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ekLW (1/2)

Thank you for your feedback. We have incorporated your comments into our revised draft (changes shown in blue) and believe they improve the paper. We address each Weakness (W) and Question (Q) below.

W1) Renewing PreScience only requires a free Semantic Scholar API key and access to the latest public arXiv snapshot. We provide the entire sequence of scripts and commands to be run for constructing the dataset in “Building the Dataset from Scratch” section of the README we provide with our released code: https://anonymous.4open.science/r/pscience-7F93/README.md

The only manual step, that you correctly pointed out, involved refining the list of topics assigned to papers. Following your feedback, we have replaced it with a fully automatic procedure that can be used identically for both domains: an LLM describes 5,000 random train-period papers in short phrases, merges these phrases into ~200 topics, and labels each target paper. Our analysis shows that SOTA LLMs come up with largely consistent topics when prompted this way (e.g., ~90% overlap in taxonomy and 84–94% agreement in assigned topics; details in Appendix B.3.1), so renewing the dataset can simply use the latest SOTA LLM with our procedure. Using this procedure, only 0.4% (AI) / 5.8% (Quantum) of papers get no topic. We show Δshare R² / Pearson results using these topics below (Tables 5 and 29). The last four rows are new methods we have added following your feedback:

BaselineAI UnaryAI PairsQuantum UnaryQuantum Pairs
Mean−15.45 / −0.04−10.53 / 0.13−53.25 / −0.10−10.59 / 0.11
Linear Extrapolation0.41 / 0.710.12 / 0.56−2.43 / 0.13−2.63 / 0.07
Citation Momentum−4.24 / 0.44−6.16 / 0.20−11.78 / 0.28−8.08 / 0.12
Author Momentum (h-index)−0.49 / 0.31−0.53 / 0.22−18.16 / −0.21−6.76 / −0.12
Chronos-Bolt [1]0.05 / 0.23−0.11 / 0.08−1.04 / 0.01−1.00 / −0.07
Chronos-2 [2]0.00 / 0.13−0.10 / 0.07−0.75 / 0.04−1.07 / −0.09
GPT-5−0.05 / 0.300.01 / 0.31−0.95 / 0.10−0.45 / 0.04
Autoresearch−0.62 / 0.09−6.83 / −0.05−3.53 / −0.08−1.93 / 0.10

On AI, the results are qualitatively similar to those observed using the original manually refined set of topics. We see that both GPT-5 and Chronos underperform relative to simple linear extrapolation. Quantum topics are smaller and noisier, and no baseline reaches positive R² there, making it a harder testbed. We found that our autoresearch run (12 hours, Claude Opus 4.7) for this task achieved an R² of 0.68 on its dev split, but −0.62 on the test split, showing significant overfitting. We have updated the paper (Section 4.4, Appendix B.3) to use only the new automatic topics for both domains, and will release the updated dataset and code with our camera-ready version.

W2) Thank you for this pointer. Connecting collaborator prediction to the classic link prediction literature is a valuable suggestion. We evaluated Common Neighbors [3], Adamic-Adar [4] and Truncated Katz [5], building each co-authorship graph only from papers dated before the target paper. Results are nDCG@1000 / R-precision:

BaselineAIQuantum
Frequency0.39 / 0.280.44 / 0.31
Common Neighbors0.33 / 0.190.35 / 0.20
Adamic-Adar0.36 / 0.220.39 / 0.23
Truncated Katz (≤3 hops, β tuned on train)0.37 / 0.260.40 / 0.28
Autoresearch0.47 / 0.330.55 / 0.38

None of these methods beat Frequency, and all trail Autoresearch, so the claim in Section 4.2 holds for collaborator prediction. We have added these to Table 3. We investigated why these methods fall below Frequency and found that they give 11–25% of their top-ranked slots to people the seed author has never co-authored with. This barely helps on first-time collaborators (hit rate ≤ 1.5%) but displaces repeat collaborators, which Frequency ranks well (hit rate 0.41–0.46 vs. 0.53 on AI). We have added this analysis to Appendix E.2.3 (Figure 9).

AuthorsOpenReview ↗

Response to Reviewer ekLW (2/2)

Q1) Please see W1. We have updated our pipeline to generate topics fully automatically following your feedback. No stage needs manual work. The only paid API is the LLM used for topic generation and labeling (about \75 AI / \35 Quantum with OpenAI Batch). Times below are estimated from measured per-request S2 latencies with a standard key:

StageAIQuantum
Targets, authors, publication lists4 h2.5 h
Influential references of targets49 h13 h
Influential references of publication-history papers15.6 d11.2 d
Author disambiguation (API features + local clustering)8 h + 1–3 d5 h + 1–2 d
Topic generation + labeling (OpenAI API)~2 h~1 h
Citation trajectories14 h10 h
Total19 d12.5 d

These times assume the standard key's introductory rate limit (1 request/s). S2 grants keys with an order of magnitude higher rate limits on request; at 10 requests/s, the full build takes about 2 d (AI) / 1.5 d (Quantum) of API time. We have added this table to Appendix B.4 (Table 10) and updated the paper. We will release the updated code with our camera-ready version.

Q2) This is a good catch. Methods only see earlier papers/authors dated before the target under consideration, but a paper/author can indeed be included in the retrieval pool due to a future target paper. However, these represent a minority. Only 4.5% (AI) / 8.9% (Quantum) of true collaborators, and 13.2% / 16.2% of all ranked candidates, are reachable solely through later target papers. To test whether this favors Autoresearch, we restrict each method's ranking to authors of earlier targets, whose full histories do not depend on later papers (targets with a collaborator in that pool; nDCG@1000):

BaselineAI: allAI: restrictedQuantum: allQuantum: restricted
Autoresearch0.470.510.550.60
GPT-5 agent0.410.460.480.53
Frequency0.390.430.440.50
Truncated Katz0.370.400.400.45

We find that this does not qualitatively change results. We have added this analysis to Appendix E.2.4 (Table 17).

Q3) Following your suggestion, we prompted GPT-5 (cutoff Sep 2024, before the test period) with each topic's name, its 12 monthly history counts, and corpus totals, and asked it to forecast test-period counts. We also added Chronos-Bolt [1] and Chronos-2 [2], and an autoresearch baseline (see W1). We have added these results to the paper (Table 5).

[1] Ansari et al. Chronos: Learning the Language of Time Series. TMLR 2024.

[2] Ansari et al. Chronos-2: From Univariate to Universal Forecasting. arXiv 2025.

[3] Newman. Clustering and Preferential Attachment in Growing Networks. Physical Review E, 2001.

[4] Adamic and Adar. Friends and Neighbors on the Web. Social Networks, 2003.

[5] Katz. A New Status Index Derived from Sociometric Analysis. Psychometrika, 1953.

Thank the authors for the detailed response and the new experiments, which addressed all of my concerns. The fully automatic topic generation, the classic link-prediction baselines, and the analysis restricted to earlier target papers are all helpful.

I have two small suggestions. First, since renewability is one of the main claims of the paper, I suggest briefly describing the renewal procedure and its cost in the main text, not only in the appendix. Second, if each renewal uses the latest LLM to generate the topics, the topics may differ between versions. It would be good to release the topic set of each version, so that results on the topic trend tasks can be compared across versions.

I'll increase my soundness score.

AuthorsOpenReview ↗

Followup response to Reviewer ekLW

Thank you for your response. We are glad we were able to address all your concerns.

Regarding your two suggestions,

  1. We agree that adding a description of the renewal procedure to the main paper is a good idea. We will make sure to include this in our camera ready draft.
  2. Regarding the topics, we do already ship a canonical set of topics generated by GPT-5.4 along with the dataset export (see Section 2, line 137 and Appendix B.2, Table 7). The cross-LLM topic generation and assignment agreement we measure in Appendix B.3.1 is meant to be read as a validation that our results are robust to the choice of LLM used.

We are glad that you liked our work and that we have addressed your concerns. We would appreciate feedback on what the gap is between the current status of our submission and what you believe a stronger submission would be.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.