- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PARR, a “Rewrite-Retrieval-Reason”-based KGC framework based on LLM reasoning. Given an incomplete triple as the query, the LLM first act as a rewriter, generating a set of sub-queries for retrieval. These set of sub-queries will then be used as seeds to retrieve similar KG triples from the training graph. The retrieval results are expected to cover the full reasoning path to complete the triple. The retrieved results is then passed back to the LLM, which will act as a reasoner, generate chain-of-thought rationales and propose the answer entity for the incomplete query triple. The LLM is trained in two steps, SFT and RL. In the SFT stage, the LLM is trained on automatically constructed trajectories. For the reasoning step, the supervision signal comes from the training triple itself and one of the reasoning paths connecting its subject and object. For the rewriting step, the signal comes from the smallest group of sub-queries that is able to obtain the entire reasoning path connecting the known entity and the golden answer. In the RL stage, two-level tree-structured rollouts are used to assign credit to rewriting and reasoning separately. Each rewrite is rewarded by its retrieval coverage of the grounding paths and the downstream accuracy of its reasoning rollouts, while each reasoning rollout is rewarded with answer accuracy, path format, and path validity. Advantages are then computed group-relatively within each level of the tree, and both segments are jointly optimized with GSPO.
Critical strengths
-
This manuscript evaluates the proposed method in different settings (transductive vs. inductive) and on different datasets (fb15k, wordnet, yago, and nell), though some of the datasets are outdated, namely fb15k and wordnet, which is less challenging for the LLM.
-
Different from prior LLM-based KGC approaches such as [1] and [2], PARR supports multi-answer supervision for both retrieval and reasoning. This explains why PARR achieves desirable hits@10 improvements.
-
Unlike prior RAG-based KGC approaches, PARR uses the LLM to rewrite the original query into multiple sub-queries and retrieves relevant context for each of them. This improves the recall of reasoning paths connecting the head entity to the target tail entity, which provides direct evidence for determining whether the triple holds.
-
The manuscript includes detailed ablation studies about retrieval methods, reward design, the effect of noisy training paths.
-
To the best of this reviewer’s knowledge, this is the first LLM-based KGC method that introduces RL-based LLM post-training, specifically GSPO.
Critical weaknesses
-
PARR is indeed not the first native LLM-based framework that significantly surpasses traditional discriminative KGC methods. The first one should be DIFT[1] accepted to ISWC 2024, KGR3 [2] and DrKGC [3] also achieve good performance. It is not necessary to compare all of them, but they should be discussed in the related work section.
-
Some of the techniques adopted in the proposed method have been widely studied. For instance, BERT-RL [4], KRST [5], APST [6], and CATS [7] utilizes relation paths connecting the head and the tail entity. It should be recognized that the retrieval method leveraged in the PARR differs from them, but their contribution should also be recognized in the related work section.
-
It is debatable whether it is necessary to leverage LLMs with advanced fine-tuning techniques for the KGC task, especially on those encyclopedia / dictionary-based datasets like FB15k and Wordnet. Modern LLMs have already been pre-trained on dictionaries and the Wikipedia. Whether the proposed method is still valuable to high-stake domains / domain-specific KGs which the LLM lacks specific knowledge remain questionable.
-
Table 1 lacks Hits@1/3 results for many baseline methods, and some baselines report only one or two performance metrics. This reviewer suggests that the authors reproduce several key baselines to make the comparison more complete.
[1] Liu et al., Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion (ISWC 2024)
[2] Li et al., Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion (NAACL 2025)
[3] Xiao et al., DrKGC: Dynamic Subgraph Retrieval-Augmented LLMs for Knowledge Graph Completion across General and Biomedical Domains (EMNLP 2025 Findings).
[4] Zha, Chen, and Yan, Inductive Relation Prediction by BERT (AAAI 2022)
[5] Su et al., Multi-aspect explainable inductive relation prediction by sentence transformer. (AAAI 2023)
[6] Su et al., Anchoring Path for Inductive Relation Prediction in Knowledge Graphs (AAAI 2024)
[7] Li et al., Context-aware Inductive Knowledge Graph Completion with Latent Type Constraints and Subgraph Reasoning (AAAI 2025)
Questions for authors
- During the rewriting phase, the LLM produces a set of sub-queries, which are in the triple format. This reviewer would like to know the relationship between each q_i in Q and the triples within the KG.
5 comments
Reply (2/3)
W3: "It is debatable whether it is necessary to leverage LLMs with advanced fine-tuning techniques for the KGC task, especially on those encyclopedia / dictionary-based datasets like FB15k and Wordnet. Modern LLMs have already been pre-trained on dictionaries and the Wikipedia. Whether the proposed method is still valuable to high-stake domains / domain-specific KGs which the LLM lacks specific knowledge remain questionable."
We would like to first note that counter-intuitively, an LLM's pretrained knowledge brings little improvement for KGC, even on datasets built from common world knowledge such as FB15k-237 and WN18RR:
- An LLM naively fine-tuned on KGC performs poorly (Tables 1 and 6; e.g., 0.155 vs. 0.387 HITS@1 on FB15k-237).
- KICGPT uses a commercial frontier LLM (ChatGPT) with far more world knowledge than PARR's 8B backbones, and PARR still outperforms it (Table 1). KG-FIT injects open-world knowledge from GPT-4o into KGE training, yet on FB15k-237 it gains at most about 3 points of HITS@1 over the strongest base models (HAKE +2.8, RotatE +3.3, TuckER +2.5), and its best variant (0.277) remains far below PARR (0.387).
- KG foundation models such as ULTRA have shown that structural knowledge matters more than semantic knowledge for KGC. This is the main reason why LLMs still struggle with KGC despite their open-world knowledge and reasoning ability. Among the many LLM-based KGC works, PARR is the first native LLM to reach superior KGC performance.
We additionally evaluate PARR on PrimeKG, a biomedical KG, using the PrimeKG split released by KG-FIT (10,344 entities, 11 relations, 100K/3K/3K triplets) with filtered HITS@k. Baseline numbers are taken from KG-FIT (Table 1).
| Model | Type | HITS@1 | HITS@5 | HITS@10 |
|---|---|---|---|---|
| PKGC | PLM-based | 0.391 | 0.565 | 0.625 |
| CSProm-KG | PLM-based | 0.492 | 0.652 | 0.745 |
| SimKGC | PLM-based | 0.524 | 0.679 | 0.742 |
| TuckER | KGE | 0.442 | 0.663 | 0.737 |
| RotatE | KGE | 0.447 | 0.646 | 0.727 |
| ConvE | KGE | 0.456 | 0.645 | 0.760 |
| DistMult | KGE | 0.475 | 0.699 | 0.782 |
| HAKE | KGE | 0.515 | 0.708 | 0.760 |
| ComplEx | KGE | 0.522 | 0.728 | 0.789 |
| KG-FIT (HAKE) | LLM-enhanced KGE | 0.605 | 0.785 | 0.835 |
| PARR | native LLM | 0.683 | 0.831 | 0.874 |
PARR outperforms the prior SOTA, KG-FIT (HAKE), by 7.8 points of HITS@1 on this biomedical KG, so its gains hold on a domain-specific KG as well. We have added these results to the paper (Appendix D, Table 14).
Reply (3/3)
W4: "Table 1 lacks Hits@1/3 results for many baseline methods, and some baselines report only one or two performance metrics. This reviewer suggests that the authors reproduce several key baselines to make the comparison more complete."
Most missing entries are metrics that the original papers do not report on these datasets. For instance, ULTRA, TRIX, KG-ICL, MOTIF, SEMMA and KRLM report MRR and HITS@10 without HITS@1/3, and many works only evaluate on FB15k-237 and WN18RR.
We are reproducing the methods that release checkpoints or a reliable codebase. Since this takes time and does not affect the main claims of the paper, we will update the table later in the discussion period once we have enough data points.
Q1: "During the rewriting phase, the LLM produces a set of sub-queries, which are in the triple format. This reviewer would like to know the relationship between each q_i in Q and the triples within the KG."
Each in an MRS target is a triplet of the training KG (a key of the precomputed retrieval map). We have clarified this in Sec. 3.3 (changes are highlighted).
Additional results from our replies to other reviewers.
We have also included additional experiments and ablations that could be potentially interesting to the reviewer. We therefore include a quick pointer below:
- Pretraining in the ULTRA setting (reply to Cz3R, W1; Appendix D, Table 15). To test the scalability and generalizability of PARR towards a KG foundational model, we trained on the same three KGs as ULTRA and TRIX (FB15k-237, WN18RR, CoDEx-M). PARR outperforms both on all nine transductive and inductive benchmarks we test, e.g., 0.759 vs. 0.676 HITS@10 on NL-100.
- Tree-structured GSPO (reply to qXKX, W1; Appendix E, Table 20). We additionally ablate the effect of with or without tree-structured GSPO, where we show it adds 2.8 / 3.1 points of HITS@1 on FB15k-237 / WN18RR.
Reply (1/3)
DIFT, KGR3 and DrKGC are reranking-based methods: they rely on a pretrained link predictor (e.g., NBFNet) to produce a candidate set (e.g., the top-20), and the LLM then outputs only the top-1 entity. Therefore:
We have updated the related work section (Appendix B) to describe these methods and how they differ from PARR.
We have added a paragraph on these methods to the related work section (Appendix B).