- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PrimeKG-CL provides six sequential tasks, multimodal node features, and persistent/removed/added test strata. The authors evaluate six continual-learning strategies across four KGE decoders, alongside LKGE, CMKL, and a Qwen-based RAG system, on link prediction, entity classification, and KGQA. They report strong decoder–strategy interactions, improved entity classification with multimodal CMKL, and a large persistent-versus-removed score gap only for DistMult.
Critical strengths
1 The benchmark captures substantial changes in a real biomedical KG, including millions of added and removed edges. The paper also documents which databases were not re-queried and discusses licensing, making the dataset’s scope relatively transparent.
2 The six-strategy by four-decoder evaluation, run over five seeds, provides useful comparisons across multiple tasks. The results show that continual-learning performance depends strongly on the KGE decoder, a practical point for future benchmark design.
Critical weaknesses
1 The five post-base tasks group additions by entity type rather than follow a chronological update stream. Appendix S6 reports marked order sensitivity for Naive-DistMult: AP is 0.096 in the canonical order, 0.072 in reverse, and 0.027 in a random order. Since the main comparisons use only the canonical order, the broader decoder–strategy conclusions may depend on this choice. The paper should foreground that the findings are conditional on this fixed sequence.
2 The “correct forgetting” interpretation in Table 5 is not established by the evaluation as described. Under the stated 70/10/20 split, the removed triples in the base-task test stratum are held out from training; lower final MRR therefore does not directly show that a model unlearned facts it had previously learned. Also, absence from the later snapshot does not establish that an association is biologically false, as the paper itself notes. Table 5 covers only three methods and two decoders, further limiting the scope of this conclusion. The result is better framed as a change in rankings across snapshot strata unless the authors clarify how it measures unlearning.
Questions for authors
Are the removed triples in Table 5 held out from training as part of the base-task test split? Do any methods receive explicit deletion or retraction signals for these triples? If not, should this analysis be described as a ranking shift on held-out snapshot strata rather than unlearning of previously learned facts?
What motivates the canonical task order? Given the order sensitivity reported in Appendix S6, are the main decoder–strategy conclusions intended to apply only to this fixed sequence?