acceptodds
Under review as a conference paper at ICLR 2027

PrimeKG-CL: A Continual Graph Learning Benchmark on Evolving Biomedical Knowledge Graphs

Abstract

Biomedical knowledge graphs underwrite drug repurposing and clinical decision support, yet the upstream ontologies they depend on (GO, HPO, MONDO, CTD) update on independent cycles that add millions of edges and deprecate hundreds of thousands more between releases. Yet existing continual graph learning (CGL) has been studied almost exclusively on synthetic random splits of static, generic KGs, a regime that cannot reproduce the asynchronous, structured evolution real biomedical KGs undergo. To this end, we introduce PrimeKG-CL, a CGL benchmark built on PrimeKG (integrating 20+ biomedical databases; 129K+ nodes, 8.1M+ edges, 10 node types, 30 relation types) with a second temporal snapshot reconstructed from nine freely re-queried databases ( June 2021, July 2023; 5.76M edges added, 889K removed, 7.21M persistent), six entity-type-grouped tasks tracking real additions, multimodal node features (BiomedBERT text, Morgan fingerprints, R-GCN structural), and a per-task persistent/removed/added test stratification. On three tasks (biomedical relationship prediction, entity classification, KGQA), we evaluate six CL strategies across four KGE decoders, plus LKGE, an LLM-RAG agent, and CMKL. We find that decoder choice dominates the continual-learning strategy: link-prediction accuracy spans roughly across decoders (RotatE DistMult ComplEx TransE) versus up to 4 across CL strategies, and the two interact so that no single strategy is best across decoders (e.g., EWC markedly benefits ComplEx while Distillation degrades RotatE). Moreover, only DistMult exhibits a clear separation between persistent and deprecated knowledge ( higher MRR on persistent vs. removed triples), indicating that standard metrics conflate retention of still-valid facts with failure to forget outdated ones; this effect is absent under RotatE. In addition, multimodal fusion improves entity classification by over feature-matched continual baselines (Macro-F1 vs. , ), and a recent CKGE framework (IncDE) failed to scale to our 5.67M-triple base task across five attempts up to 350 GB RAM. Data, pipeline, baselines, and the stratified split will be released openly.

Primary area:
learning on time series and dynamical systems
Keywords:
Continual Learning, Biomedical Knowledge Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a2piOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrimeKG-CL provides six sequential tasks, multimodal node features, and persistent/removed/added test strata. The authors evaluate six continual-learning strategies across four KGE decoders, alongside LKGE, CMKL, and a Qwen-based RAG system, on link prediction, entity classification, and KGQA. They report strong decoder–strategy interactions, improved entity classification with multimodal CMKL, and a large persistent-versus-removed score gap only for DistMult.

Critical strengths

1 The benchmark captures substantial changes in a real biomedical KG, including millions of added and removed edges. The paper also documents which databases were not re-queried and discusses licensing, making the dataset’s scope relatively transparent.

2 The six-strategy by four-decoder evaluation, run over five seeds, provides useful comparisons across multiple tasks. The results show that continual-learning performance depends strongly on the KGE decoder, a practical point for future benchmark design.

Critical weaknesses

1 The five post-base tasks group additions by entity type rather than follow a chronological update stream. Appendix S6 reports marked order sensitivity for Naive-DistMult: AP is 0.096 in the canonical order, 0.072 in reverse, and 0.027 in a random order. Since the main comparisons use only the canonical order, the broader decoder–strategy conclusions may depend on this choice. The paper should foreground that the findings are conditional on this fixed sequence.

2 The “correct forgetting” interpretation in Table 5 is not established by the evaluation as described. Under the stated 70/10/20 split, the removed triples in the base-task test stratum are held out from training; lower final MRR therefore does not directly show that a model unlearned facts it had previously learned. Also, absence from the later snapshot does not establish that an association is biologically false, as the paper itself notes. Table 5 covers only three methods and two decoders, further limiting the scope of this conclusion. The result is better framed as a change in rankings across snapshot strata unless the authors clarify how it measures unlearning.

Questions for authors

Are the removed triples in Table 5 held out from training as part of the base-task test split? Do any methods receive explicit deletion or retraction signals for these triples? If not, should this analysis be described as a ranking shift on held-out snapshot strata rather than unlearning of previously learned facts?

What motivates the canonical task order? Given the order sensitivity reported in Appendix S6, are the main decoder–strategy conclusions intended to apply only to this fixed sequence?

Reviewer_SvU1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PrimeKG-CL, a continual graph learning benchmark built on a biomedical knowledge graph. The June 2021 release of PrimeKG serves as t₀. The authors re-query July 2023 versions of nine freely available source databases to reconstruct a t₁ snapshot. They then group the edges added between the two snapshots into five entity-type tasks, which together with the full t₀ graph (Base) form a six-task sequence. The Base test set is further stratified into persistent and removed triples, according to whether each triple still exists at t₁. Textual, molecular, and structural node features are provided. The experiments cover link prediction, node classification, and KGQA. They evaluate six CL strategies combined with four KGE decoders, together with LKGE, CMKL, and a RAG agent.

Critical strengths
  1. Building continual-learning tasks from real knowledge-base updates is a worthwhile direction. Using the persistent/removed stratification to separate knowledge that should be retained from knowledge that should be forgotten is a question worth studying.
  2. The data release is well documented. Restrictively licensed sources are not redistributed but can be rebuilt with a provided script, and the release includes a datasheet, Croissant metadata, and a maintenance plan.
Critical weaknesses
  1. The evaluation protocol does not adequately establish that the benchmark measures temporal continual learning on evolving biomedical knowledge. a) The task order is imposed, not observed. The benchmark rests on genuine snapshot differences. However, there are only two real time points, and the sequential protocol imposes a manual ordering on entity-type groups derived from a single t₀→t₁ transition. Appendix S6 shows that changing the task order alters AP by up to about 3.5× (0.096, 0.072, 0.027), and the authors acknowledge that the canonical order is a favorable case. Real temporal change and an artificial task sequence are thus conflated. b) The update is incomplete. The t₁ snapshot refreshes only 9 of the 16 source databases. The other seven, including all drug-related sources, are carried forward unchanged from t₀. The introduction uses drug repurposing as a central motivation, yet the benchmark contains no drug-related evolution. c) Source diversity is limited. About 95% of the added edges come from a single bulk refresh of Bgee. The paper claims to capture asynchronous, structured evolution across many databases, but it does not show that its main findings hold when the Bgee additions are excluded or when sources are evaluated separately.
  2. The adequacy of optimization has not been established, and some results need verification. a) Training budget and convergence. According to Appendix S1, each epoch samples only 50,000 triples, so 100 epochs amount to about 5M draws. This is fewer than the roughly 5.67M training triples in the Base task. If sampling is with replacement, only about 59% of distinct triples may be seen even once. No convergence curves or coverage analysis are provided. b) No standardized optimization protocol. The authors run some sensitivity analyses (the EWC coefficient, the replay buffer size, and the learning rate for Naive-ComplEx), but the main comparisons rely on shared training settings. The paper does not show that each decoder received a comparable hyperparameter search. Two observations are hard to interpret without such evidence: TransE reaches an AP of only 0.007, far below the other decoders; on DistMult, Joint Training (the oracle) scores below Naive (0.078 vs 0.096). As a result, the roughly 20× gap across decoders cannot yet be separated from the effect of the training configuration. Prior work has shown that the training setup often matters more than the choice of KGE model (Ruffinelli et al., 2020). c) Results that need checking. In Table 4, SI, Distillation, and MIR, three methods with entirely different mechanisms, report identical means and standard deviations for AP, AF, and BWT (0.362 ± 0.004, 0.007 ± 0.007, 0.015 ± 0.006). The authors should check whether results were reused, whether the implementations collapse to the same configuration, or whether the table contains a transcription error. In Table 3, Experience Replay and MIR are also nearly identical, which is less surprising because MIR is a replay variant. Since Appendix S1 reports that four scoring-function bugs were fixed during revision, verifying the reported numbers is all the more important.
  3. The “correct forgetting” claim lacks longitudinal evidence. Table 5 reports only the final persistent/removed MRR ratio. This cannot distinguish between two explanations: a) the model selectively forgets the removed edges during continual training; b) the removed edges are intrinsically harder to predict. For instance, about 18% of them lose an endpoint entity at t₁. The authors should report the ratio immediately after training on Base, as well as the persistent and removed MRR trajectories across all six tasks. The stratified evaluation also covers only six configurations. Because “removed” does not imply biologically false, as the authors themselves note, this metric is better interpreted as adaptation to outdated database records than as correction of erroneous knowledge.
  4. The evaluation tracks and baselines are incomplete. a) Recent methods are missing. I acknowledge the authors’ attempts to run IncDE, which failed with out-of-memory errors and deadlocks, and the protocol incompatibilities they report for FastKGE and SAGE. Still, the benchmark currently provides no results for any recent continual KGE method, and LKGE is run under its own protocol, which the authors concede is not comparable. This limits its value as a reference for the community. Results for at least one recent method, even on a subgraph, would help. b) The purpose of the KGQA track is unclear. Appendix S5 describes the setup: 200 questions per task, Qwen2.5-7B, ChromaDB retrieval, and three configurations. However, details on question construction, answer normalization, retrieval recall, and error analysis are limited. All configurations reach a token F1 of at most 0.015, so the track does not yet discriminate between methods. More fundamentally, knowledge updates happen by replacing the retrieval index while the LLM is never trained continually, so near-zero forgetting holds by construction (Appendix S5 acknowledges this). The track therefore evaluates dynamic retrieval over an evolving graph rather than continual learning. c) The node classification setup is questionable. The task is to predict entity types (gene, disease, drug, and so on). Entity descriptions usually reveal the type directly, yet a text-only classifier reaches only 0.246 Macro-F1, which is anomalous.

Minor issues

  1. The provenance and role of CMKL are unclear. It is listed as a baseline, but its full architecture combines several existing components, and the paper does not say whether it is a new method proposed by the authors or a reproduction or adaptation of prior work. Because CMKL is the best method on node classification, this ambiguity affects how the contributions should be read.
  2. A paired t-test over five seeds yielding t = 77 shows only that the results are stable across seeds, not that the conclusions generalize.
Questions for authors
  1. Why do SI, Distillation, and MIR produce identical results in Table 4?
  2. Can you provide training convergence curves, and explain why Joint Training scores below Naive on DistMult?
  3. What is the persistent/removed MRR ratio immediately after training on Base? Can you show the MRR trajectories of both strata across the full task sequence?
  4. Do the main findings hold when the Bgee additions are excluded, or when results are reported separately by source database?
  5. How are the KGQA questions constructed, and how are answers normalized and matched? Given that knowledge is updated by replacing the index, which continual-learning capability is this track meant to evaluate?
  6. Can you report results for at least one recent continual KGE method, even on a subgraph?
  7. What determines the order of the six tasks? Given how sensitive the results are to ordering, what standard evaluation protocol should the benchmark prescribe?
  8. Is CMKL a method proposed by the authors, or an existing method?
Reviewer_PXMFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PrimeKG-CL is a continual graph learning benchmark. It takes PrimeKG as t0t_0 and rebuilds a t1t_1 snapshot by re-querying nine of its source databases (5.76M edges added, 889K removed). The stream has six tasks: the full t0t_0 graph, followed by the t1t_1 additions grouped by entity type. It ships text, molecular and structural node features, plus a persistent/removed/added test split. Six continual-learning (CL) strategies are run with four embedding decoders, together with LKGE, CMKL and a RAG agent, on link prediction, entity-type classification and KGQA. The main claims are:

  • The choice of decoder matters more than the CL strategy (~20× vs. ~4×).
  • Decoders and CL strategies interact.
  • Only DistMult separates persistent facts from removed ones.
Critical strengths
  • Real temporal evolution at scale, with a persistent/removed split. Current continual KG benchmarks use synthetic splits of graphs with ≤15K entities, which Zhao et al. (2025) show hides real shifts between snapshots. A 129K-entity graph built from real database updates, with a test split that separates still-valid facts from deprecated ones, would be a useful resource. The release plan (datasheet, licensing, maintenance) is also solid. The benchmark resource is the paper's main value.
Critical weaknesses
  • The benchmark doesn't capture the evolution it claims to.
    • About 95% of the additions come from a single Bgee expression-atlas refresh, and there are only two snapshots. The "asynchronous evolution" is therefore mostly one bulk update.
    • Tasks are ordered by entity type, not by time. So the stream is really a structured partition of one t0→t1t_0 \to t_1 diff.
    • Results depend heavily on task order: AP is 0.096 in the canonical order and 0.027 in a random order, and the authors acknowledge the canonical order is favorable.
    • The stream contains only additions; no removal events are ever given to the model. The "correct forgetting" metric therefore measures incidental drift, not something a method could learn. This undercuts the paper's main argument for the stratified split.
  • The baselines look undertrained, which puts the headline finding in doubt.
    • Training uses 50K sampled triples per epoch for 100 epochs, about 5M triples. That is less than one pass over the ~5.67M-triple base training set.
    • TransE sits at the random-ranking floor, which the authors admit.
    • ComplEx (0.018) scores far below DistMult (0.096), even though ComplEx includes DistMult as a special case.
    • The Joint Training "oracle" scores below Naive on DistMult.
    • Scoring bugs were fixed in the middle of the revision.
    • The 20× decoder spread depends on TransE being at the floor. Without TransE, RotatE/DistMult is only 1.45×, while CL strategies within the RotatE column span 2.5× (0.059–0.146). So the claim that "decoder dominates CL strategy" does not hold up.
    • The other two tracks add little. KGQA F1 is ≤0.015, and node classification predicts entity type, which is already known when the graph is built.
Questions for authors
  1. What is the total training exposure per task? With per-decoder tuning to convergence, do the decoder ranking, the ComplEx < DistMult gap and the Joint < Naive result still hold?
  2. Do the decoder × CL-strategy conclusions survive the reversed and random task orders?
  3. Does any task contain removal events? If not, how is a method supposed to achieve "correct forgetting"?
  4. The Gene/Protein task is described as protein–protein interactions, but the PPI sources were not re-queried. Which edges does this task actually contain? Please give per-task edge counts by source database.
  5. In Table 4, SI, Distillation and MIR have identical means and standard deviations (0.362 ± 0.004, …). Is this a bug?
  6. CMKL has no citation. Is it the authors' own method? If so, the +43% node-classification result should be presented as a method contribution, not as a baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.