acceptodds
Withdrawn from ICLR 2027

LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting the future impact of a newly published paper is challenging because it must be inferred from heterogeneous evidence available at publication time. Existing approaches often rely on a single source of information or combine multiple sources without accounting for their different predictive roles. In this paper, we present LLM4Impact, an evidence-aware method for scientific impact prediction that learns to represent, integrate, and calibrate heterogeneous information. LLM4Impact combines semantic, graph, LLM, and temporal representations, and injects graph information into a frozen LLM through continuous prefix tokens. A context aware gating mechanism adaptively weights different evidence, while a separate calibration module accounts for domain and temporal variation in citation scales. We further construct a large-scale benchmark dataset with 2 million papers, leakage-safe point-in-time heterogeneous ego graphs, temporal splits, and both year-level and month-level citation targets. Experiments show that LLM4Impact consistently outperforms strong semantic, graph, and LLM based baselines, with a 10.13% reduction in year RMSE on the in distribution test set and a 6.87% reduction under out-of-domain distribution. Our results reveal that the value of such evidence is context dependent: different papers benefit from different sources, while domain and publication time affect how evidence translates into citations. This finding motivates adaptive evidence selection and context-conditioned calibration rather than simply richer representations. We will release our code, benchmark, and an interactive web demonstration upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Impact Prediction, AI for Science, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FJG1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LLM4Impact to predict future citation counts using paper text, citation relationships, research domain, and publication time. Specifically, It combines paper text, citation graphs, and language model representations to predict future citation counts for newly published papers. It uses research domain and publication time to weight these sources and calibrate predictions. It also introduces a dataset of two million papers with yearly and monthly prediction targets.

Critical strengths

The dataset is the clearest contribution. Its scale, publication-time citation graphs, and yearly and monthly labels could be useful for future citation prediction research.

Critical weaknesses
  • The practical value of the task is unclear. The paper mentions identifying emerging research and supporting research evaluation, but does not validate either use case. It remains unclear what decisions would improve with more accurate citation predictions.

  • The methodological novelty is limited. The paper combines established text encoders, graph models, and language models, but does not clearly explain what we learn beyond improved prediction scores.

  • The evidence for the core design choices is incomplete. Temporal encoding and adaptive fusion are emphasized, but direct comparisons against simpler alternatives are missing.

  • The paper states that the Qwen models were evaluated via both prompting and fine-tuning, but It seems Tables 2 and 3 report only zero-shot results. A key control is missing. The same Qwen2.5-7B backbone fine-tuned with LoRA on title and abstract alone.

Questions for authors

How do the scope and experiments of this study support its claimed practical uses? Which results show that more accurate citation prediction helps people make better decisions?

Reviewer_VdVqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LLM4Impact, a framework for scientific impact prediction that integrates paper semantics, heterogeneous academic graphs, and temporal information. It also introduces a large-scale benchmark containing 2 million papers with yearly and monthly citation prediction targets. The experimental results are promising, and the dataset contribution is particularly valuable.

Critical strengths
  1. Large-scale benchmark contribution. The paper constructs a substantial benchmark containing 2 million papers, with heterogeneous academic graphs and both yearly and monthly citation prediction targets. The dataset and associated construction pipeline could provide a valuable resource for future research.

  2. Comprehensive statistical analysis. The paper provides extensive dataset statistics, temporal evaluation, ablation studies, and controlled perturbation experiments. These analyses offer useful insights into the roles of different information sources in scientific impact prediction.

  3. Promising predictive performance. LLM4Impact achieves consistent improvements over the reported baselines across multiple prediction horizons and temporal splits. Although the method leverages richer information than text-based approaches such as NAIP, the results suggest that incorporating heterogeneous academic information can benefit scientific impact prediction.

Critical weaknesses
  1. Insufficient isolation of semantic contributions. The paper does not clearly quantify the incremental value of the target paper's title and abstract beyond external metadata and scholarly network information. A controlled comparison between a metadata-and-graph-only model and the full model is missing.

  2. Limited gains from graph-to-LLM prefix injection. Removing the graph prefix increases RMSE by only 0.0033 on the previous-test split, compared with 0.0482 when removing the entire graph branch. This raises questions about the necessity of the proposed prefix-injection mechanism and its advantage over simpler alternatives.

  3. Insufficient validation of adaptive gating. The paper emphasizes dynamically selecting evidence sources for different papers, but does not adequately compare the proposed gating mechanism against simpler alternatives, such as fixed-weight fusion, feature concatenation, or an MLP. Consequently, the contribution of adaptive gating remains unclear.

  4. Presentation and layout issues. Some figures, particularly Figures 2 and 4, contain text that is too small to read comfortably. In addition, Table 11 on page 22 leaves a substantial amount of unused space on the right, resulting in an unbalanced layout.

Questions for authors

Have the authors explored using a JEV-style approach for scientific impact prediction? If so, how does it compare with LLM4Impact under the same input information, data splits, and prediction targets?

Reviewer_cdvnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LLM4Impact for cold-start citation prediction, combining SciBERT, a heterogeneous GNN, continuous temporal encoding, and a LoRA-tuned LLM (via soft graph prefixes) using context-aware gated fusion and affine calibration. Alongside the architecture, the authors contribute a 2-million-paper Semantic Scholar benchmark (2005 to 2020) with leakage-safe point-in-time ego-graphs for both yearly and monthly citation forecasting. They validate their approach against text, graph, and LLM baselines on in-distribution and temporally shifted splits, supported by leave-one-out module ablations, perturbation studies, and an interactive demo.

Critical strengths
  1. The paper builds a large-scale benchmark with point-in-time ego-graphs that truncate neighbor citation counts to each paper's publication year, a concrete step against temporal leakage and a potentially useful resource for cold-start impact prediction.
  2. The multi-branch design brings text, graph, LLM, and temporal signals into a single framework, and the gains over single-source baselines suggest these signals are complementary.
  3. Evaluating on both an in-distribution split and a temporally shifted split with unseen publication years is a sensible design that goes beyond standard random splits.
  4. The ablations on both splits, paired bootstrap tests, and perturbation studies help clarify how the LLM, graph, and calibration components contribute.
Critical weaknesses
  1. The paper equates raw citation counts with scientific impact, but its perturbation studies suggest the model largely tracks venue: on the same 20 CS papers, changing only the venue shifts predicted five-year citations more than any other single change, including replacing the abstract with a generic template or another paper's abstract, while removing highly cited references or adding top-cited authors barely moves predictions (Figures 4, 11, and 12). The paper reads these results as validating its multi-source design, but they show a single metadata cue dominating. Since research evaluation is a stated application, the paper should separate impact from venue visibility.

  2. The "leakage-safe" claim misses the largest risks. The LLM backbones were pretrained on web data through at least 2024, so the 2018 to 2020 fresh test is not unseen for them, which is the lookahead bias documented in LLM-based forecasting (Sarkar & Vafa, 2024). The zero-shot check in Section 3.1 does not rule this out: it omits Qwen2.5-7B, the headline backbone; zero-shot Qwen3.5-9B actually beats NAIP, H2CGL, and DPPDCC on previous-test NDCG@20; and a fine-tuned head can read knowledge from hidden states that generation does not express (cf. Gurnee & Tegmark, 2024). S2AG also merges preprint and published versions, so later venues and references can enter at "publication time".

  3. The evaluation does not show that the proposed design drives the gains. Without calibration the model is worse than NAIP on fresh test (0.868 vs 0.844); without the LLM branch it is no better than H2CGL (0.799 vs 0.796); graph-to-LLM prefix injection changes RMSE by under 0.5% (Table 9); and the temporal encoder, gate, SciBERT branch, and auxiliary losses are never ablated. There are also no simple baselines (field, venue, or year means; gradient-boosted trees on bibliometric features) and no naive fusion of the same inputs, while NAIP sees only the title and abstract.

  4. The claimed robustness to temporal shift is not supported. From in-distribution to fresh test, the model's RMSE degrades more than that of NAIP, SciBERT, or DPPDCC, so its lead over NAIP shrinks. The periodic encoder in Eq. 3 has no mechanism to extrapolate the upward citation trend the paper documents (Figure 6), and Figure 4a is consistent with this: moving the publication year to the latest year lowers the 7B model's prediction by 31.6 citations and leaves the 0.6B model essentially unchanged (+0.5).

  5. The month-level task, presented as a key contribution, is weakly motivated. About 90% of labels are zero, early counts plausibly reflect S2AG dating and preprint artifacts more than uptake, and absolute QWK is low for every method, so large relative gains say little.

Questions for authors
  1. Can you test for lookahead directly, for example with a pretraining-data detector such as Min-K% Prob (Shi et al., 2024) on test abstracts, by evaluating the month-level task on papers published after the backbone's training cutoff as forecasting benchmarks do (Karger et al., 2025), or by using a point-in-time model such as DatedGPT (Yan et al., 2026)?
  2. What do the learned gate weights look like across fields and years? Since the fusion MLP also sees all branch embeddings directly, what happens if the gate is replaced with plain concatenation?
  3. Can you compare the continuous temporal encoder with the lookup embedding on fresh test? Why does Eq. 3 omit the linear term shown in Figure 3, and how does a periodic time embedding let calibration help more on fresh test than in-distribution?
  4. Can you ablate the SciBERT branch (keeping it for node initialization) and each auxiliary loss? Was the contrastive loss active in the main results, given that Appendix B sets λcl=0\lambda_{\text{cl}} = 0 while Table 10 lists 0.5?
  5. Which configuration produced Tables 2 and 3: 1-hop or 2-hop graphs (Section 3.1 vs Table 10), GCN or GAT (Figure 3 vs Appendix F), and were the retrieval, track-record, and explicit-feature branches from Appendix B active? Which model produced Table 5, whose per-horizon values match neither backbone and whose average does not match its entries?
  6. In Figure 4a, what year does "latest" refer to, and why do the two backbones respond so differently (+0.5 vs -31.6)? Do author nodes carry any point-in-time features, given that adding two top-cited authors barely changes predictions? Do these findings hold beyond 20 CS papers, given that about three quarters of the benchmark is tagged Medicine?
  7. Removing highly cited references barely changes predictions despite the citation-weighted GNN. How much of the graph branch's ablation gain remains if venue nodes are removed? Please also confirm the identical venue effect for H2CGL and the 7B model (-89.0) in Figure 4a.
  8. Can you report ranking quality within venue, for example Spearman correlation among papers from the same venue and year? This would show whether the model captures anything beyond venue.

References for Weaknesses section and Questions sections:-

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.