acceptodds
Under review as a conference paper at ICLR 2027

Learning Generative Knowledge Graph Completion via Path Supervision from Link Predictors

Abstract

We propose Path-Augmented Retrieval and Reasoning (PARR), a framework for training generative language models for Knowledge Graph Completion (KGC). A unique challenge in adopting native LLM for KGC is the lack of retrieval and reasoning supervision from KGC datasets, which results in all previous efforts to underperform traditional KGC methods by a significant margin. In this work, we propose to leverage grounding paths sampled from interpretable link predictors to provide effective supervision for LLM-based KGC. Specifically, we utilize paths (1) to augment retrieval for enhanced sub-graph retrieval, (2) as ground-truth retrieval signals to supervise Rewriter LLM for KG-based query rewritings, and (3) as a means to "distill" the structural knowledge to a Reasoner LLM through chain-of-thoughts. To further improve reasoning capacity and reduce cascaded error, we devise a RL-based finetuning stage with soft verifiable rewards tailored for KGC. PARR is the first native LLM-based framework that surpasses traditional discriminative KGC methods, achieves superior performance compared to existing state-of-the-arts across a wide range of KG datasets under both transductive and inductive settings, while being generalizable, scalable, and interpretable.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Link prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hck1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARR, a “Rewrite-Retrieval-Reason”-based KGC framework based on LLM reasoning. Given an incomplete triple as the query, the LLM first act as a rewriter, generating a set of sub-queries for retrieval. These set of sub-queries will then be used as seeds to retrieve similar KG triples from the training graph. The retrieval results are expected to cover the full reasoning path to complete the triple. The retrieved results is then passed back to the LLM, which will act as a reasoner, generate chain-of-thought rationales and propose the answer entity for the incomplete query triple. The LLM is trained in two steps, SFT and RL. In the SFT stage, the LLM is trained on automatically constructed trajectories. For the reasoning step, the supervision signal comes from the training triple itself and one of the reasoning paths connecting its subject and object. For the rewriting step, the signal comes from the smallest group of sub-queries that is able to obtain the entire reasoning path connecting the known entity and the golden answer. In the RL stage, two-level tree-structured rollouts are used to assign credit to rewriting and reasoning separately. Each rewrite is rewarded by its retrieval coverage of the grounding paths and the downstream accuracy of its reasoning rollouts, while each reasoning rollout is rewarded with answer accuracy, path format, and path validity. Advantages are then computed group-relatively within each level of the tree, and both segments are jointly optimized with GSPO.

Critical strengths
  1. This manuscript evaluates the proposed method in different settings (transductive vs. inductive) and on different datasets (fb15k, wordnet, yago, and nell), though some of the datasets are outdated, namely fb15k and wordnet, which is less challenging for the LLM.

  2. Different from prior LLM-based KGC approaches such as [1] and [2], PARR supports multi-answer supervision for both retrieval and reasoning. This explains why PARR achieves desirable hits@10 improvements.

  3. Unlike prior RAG-based KGC approaches, PARR uses the LLM to rewrite the original query into multiple sub-queries and retrieves relevant context for each of them. This improves the recall of reasoning paths connecting the head entity to the target tail entity, which provides direct evidence for determining whether the triple holds.

  4. The manuscript includes detailed ablation studies about retrieval methods, reward design, the effect of noisy training paths.

  5. To the best of this reviewer’s knowledge, this is the first LLM-based KGC method that introduces RL-based LLM post-training, specifically GSPO.

Critical weaknesses
  1. PARR is indeed not the first native LLM-based framework that significantly surpasses traditional discriminative KGC methods. The first one should be DIFT[1] accepted to ISWC 2024, KGR3 [2] and DrKGC [3] also achieve good performance. It is not necessary to compare all of them, but they should be discussed in the related work section.

  2. Some of the techniques adopted in the proposed method have been widely studied. For instance, BERT-RL [4], KRST [5], APST [6], and CATS [7] utilizes relation paths connecting the head and the tail entity. It should be recognized that the retrieval method leveraged in the PARR differs from them, but their contribution should also be recognized in the related work section.

  3. It is debatable whether it is necessary to leverage LLMs with advanced fine-tuning techniques for the KGC task, especially on those encyclopedia / dictionary-based datasets like FB15k and Wordnet. Modern LLMs have already been pre-trained on dictionaries and the Wikipedia. Whether the proposed method is still valuable to high-stake domains / domain-specific KGs which the LLM lacks specific knowledge remain questionable.

  4. Table 1 lacks Hits@1/3 results for many baseline methods, and some baselines report only one or two performance metrics. This reviewer suggests that the authors reproduce several key baselines to make the comparison more complete.

[1] Liu et al., Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion (ISWC 2024)

[2] Li et al., Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion (NAACL 2025)

[3] Xiao et al., DrKGC: Dynamic Subgraph Retrieval-Augmented LLMs for Knowledge Graph Completion across General and Biomedical Domains (EMNLP 2025 Findings).

[4] Zha, Chen, and Yan, Inductive Relation Prediction by BERT (AAAI 2022)

[5] Su et al., Multi-aspect explainable inductive relation prediction by sentence transformer. (AAAI 2023)

[6] Su et al., Anchoring Path for Inductive Relation Prediction in Knowledge Graphs (AAAI 2024)

[7] Li et al., Context-aware Inductive Knowledge Graph Completion with Latent Type Constraints and Subgraph Reasoning (AAAI 2025)

Questions for authors
  1. During the rewriting phase, the LLM produces a set of sub-queries, which are in the triple format. This reviewer would like to know the relationship between each q_i in Q and the triples within the KG.
5 comments
AuthorsOpenReview ↗

Reply (1/3)

W1: "PARR is indeed not the first native LLM-based framework that significantly surpasses traditional discriminative KGC methods. The first one should be DIFT[1] accepted to ISWC 2024, KGR3 [2] and DrKGC [3] also achieve good performance. It is not necessary to compare all of them, but they should be discussed in the related work section."

DIFT, KGR3 and DrKGC are reranking-based methods: they rely on a pretrained link predictor (e.g., NBFNet) to produce a candidate set (e.g., the top-20), and the LLM then outputs only the top-1 entity. Therefore:

  • These methods depend on a separately trained link predictor at inference, and some of them also modify LLM architecture (DIFT injects KG embeddings through a trainable projector, and DrKGC adds a GCN adapter). They are therefore not native LLM-based KGC in the sense of our paper. We have added a footnote in the revised footnote on the first page: the LLM keeps its architecture and generates the predictions itself.
  • They use the LLM for reranking, a much simpler task than generating the top-10 candidates directly as PARR does. Even within this simpler task, the LLM selects only the top-1 entity. KGR3 itself notes that its HITS@k is "implicitly constrained by the Hits@n performance of the base KGC model".
  • Reranking is expected to improve over retrieval, especially when the top-1 is picked from a larger candidate set (e.g., the top-20 in KGR3). In embedding models and recommender systems, it is common practice not to compare rerankers with retrievers, since they belong to different stages (for instance, PARR can be easily combined with any of these methods).

We have updated the related work section (Appendix B) to describe these methods and how they differ from PARR.

W2: "Some of the techniques adopted in the proposed method have been widely studied. For instance, BERT-RL [4], KRST [5], APST [6], and CATS [7] utilizes relation paths connecting the head and the tail entity. It should be recognized that the retrieval method leveraged in the PARR differs from them, but their contribution should also be recognized in the related work section."

  • BERT-RL, KRST, APST and CATS use paths found by heuristic graph search as context for scoring a given candidate triplet (h, r, t). This is similar to one of the baseline non-learned retrievers we included. We have shown that our rewriting-based retrieval trained with Minimal Rewriting Sets significantly outperforms such baseline (Figure 3). These methods also do not use paths as supervision for retrieval or as CoT signals, so they differ substantially from PARR.
  • Our additional ablation (see our reply to Reviewer Cz3R, W2) shows that paths from simple graph traversal degrade PARR in both SFT and RL (e.g., FB15k-237 HITS@1 drops from 0.348 to 0.241 after SFT and from 0.387 to 0.255 after RL). This shows that extracting grounding paths from pretrained link predictors is necessary.

We have added a paragraph on these methods to the related work section (Appendix B).

AuthorsOpenReview ↗

Reply (2/3)

W3: "It is debatable whether it is necessary to leverage LLMs with advanced fine-tuning techniques for the KGC task, especially on those encyclopedia / dictionary-based datasets like FB15k and Wordnet. Modern LLMs have already been pre-trained on dictionaries and the Wikipedia. Whether the proposed method is still valuable to high-stake domains / domain-specific KGs which the LLM lacks specific knowledge remain questionable."

We would like to first note that counter-intuitively, an LLM's pretrained knowledge brings little improvement for KGC, even on datasets built from common world knowledge such as FB15k-237 and WN18RR:

  • An LLM naively fine-tuned on KGC performs poorly (Tables 1 and 6; e.g., 0.155 vs. 0.387 HITS@1 on FB15k-237).
  • KICGPT uses a commercial frontier LLM (ChatGPT) with far more world knowledge than PARR's 8B backbones, and PARR still outperforms it (Table 1). KG-FIT injects open-world knowledge from GPT-4o into KGE training, yet on FB15k-237 it gains at most about 3 points of HITS@1 over the strongest base models (HAKE +2.8, RotatE +3.3, TuckER +2.5), and its best variant (0.277) remains far below PARR (0.387).
  • KG foundation models such as ULTRA have shown that structural knowledge matters more than semantic knowledge for KGC. This is the main reason why LLMs still struggle with KGC despite their open-world knowledge and reasoning ability. Among the many LLM-based KGC works, PARR is the first native LLM to reach superior KGC performance.

We additionally evaluate PARR on PrimeKG, a biomedical KG, using the PrimeKG split released by KG-FIT (10,344 entities, 11 relations, 100K/3K/3K triplets) with filtered HITS@k. Baseline numbers are taken from KG-FIT (Table 1).

ModelTypeHITS@1HITS@5HITS@10
PKGCPLM-based0.3910.5650.625
CSProm-KGPLM-based0.4920.6520.745
SimKGCPLM-based0.5240.6790.742
TuckERKGE0.4420.6630.737
RotatEKGE0.4470.6460.727
ConvEKGE0.4560.6450.760
DistMultKGE0.4750.6990.782
HAKEKGE0.5150.7080.760
ComplExKGE0.5220.7280.789
KG-FIT (HAKE)LLM-enhanced KGE0.6050.7850.835
PARRnative LLM0.6830.8310.874

PARR outperforms the prior SOTA, KG-FIT (HAKE), by 7.8 points of HITS@1 on this biomedical KG, so its gains hold on a domain-specific KG as well. We have added these results to the paper (Appendix D, Table 14).

AuthorsOpenReview ↗

Reply (3/3)

W4: "Table 1 lacks Hits@1/3 results for many baseline methods, and some baselines report only one or two performance metrics. This reviewer suggests that the authors reproduce several key baselines to make the comparison more complete."

Most missing entries are metrics that the original papers do not report on these datasets. For instance, ULTRA, TRIX, KG-ICL, MOTIF, SEMMA and KRLM report MRR and HITS@10 without HITS@1/3, and many works only evaluate on FB15k-237 and WN18RR.

We are reproducing the methods that release checkpoints or a reliable codebase. Since this takes time and does not affect the main claims of the paper, we will update the table later in the discussion period once we have enough data points.

Q1: "During the rewriting phase, the LLM produces a set of sub-queries, which are in the triple format. This reviewer would like to know the relationship between each q_i in Q and the triples within the KG."

Each qiq_i in an MRS target is a triplet of the training KG (a key of the precomputed retrieval map). We have clarified this in Sec. 3.3 (changes are highlighted).

Additional results from our replies to other reviewers.

We have also included additional experiments and ablations that could be potentially interesting to the reviewer. We therefore include a quick pointer below:

  • Pretraining in the ULTRA setting (reply to Cz3R, W1; Appendix D, Table 15). To test the scalability and generalizability of PARR towards a KG foundational model, we trained on the same three KGs as ULTRA and TRIX (FB15k-237, WN18RR, CoDEx-M). PARR outperforms both on all nine transductive and inductive benchmarks we test, e.g., 0.759 vs. 0.676 HITS@10 on NL-100.
  • Tree-structured GSPO (reply to qXKX, W1; Appendix E, Table 20). We additionally ablate the effect of with or without tree-structured GSPO, where we show it adds 2.8 / 3.1 points of HITS@1 on FB15k-237 / WN18RR.
Reviewer_Cz3ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARR, a framework for training a native generative LLM for knowledge graph completion (KGC). Motivated by the lack of retrieval and reasoning supervision in KGC datasets, PARR uses grounding paths extracted from pretrained interpretable link predictors as supervision. A single LLM performs a rewrite–retrieve–reason process, in which it decomposes a query into sub-queries for retrieval and then generates chain-of-thought reasoning paths and answers over the retrieved triplets. The paths are used to augment retrieval, to construct minimal rewriting sets via set cover as rewriting targets, and as chain-of-thought targets for reasoning. The model is first trained with SFT and then trained end-to-end with GSPO over tree-structured rollouts, using rewards based on answer accuracy, path format and validity, and path coverage. Experiments on 19 KGs under transductive and inductive settings show that PARR outperforms existing LLM-based and traditional discriminative KGC methods, including the link predictors that supply its supervision.

Critical strengths

S1. Strong empirical performance across diverse KGs.

PARR achieves state-of-the-art performance on most of the 19 evaluated KGs, which range from small statistical KGs to the large-scale ogbl-wikikg2, under transductive, inductive (unseen entities), and fully inductive (unseen entities and relations) settings. Notably, it surpasses the link predictors that supply its own supervision (NBFNet and A*Net), and clearly outperforms methods based on commercial LLMs such as ChatGPT (e.g., GPT-3.5 Turbo and KICGPT).

S2. Novel use of link-predictor paths as supervision.

Using grounding paths extracted from pretrained link predictors as supervision for LLM retrieval and reasoning is a novel idea that addresses the lack of such supervision in KGC datasets. The paths are used consistently across the pipeline, serving as targets for query rewriting via minimal rewriting sets and as chain-of-thought targets for reasoning. In addition, the subsequent RL stage with verifiable rewards allows the model to go beyond the extracted paths, which is an interesting combination for link prediction.

Critical weaknesses

W1. The claimed benefits of the generative formulation are not demonstrated.

The title and introduction motivate PARR by its generative capability and present it as an advantage over discriminative link predictors. However, the experiments are limited to single-task link prediction on each KG separately, so they do not show whether the claimed strengths (e.g., generalizability and conversational use) actually materialize. In particular, PARR requires training link predictors, extracting paths, and running SFT and RL anew for each target KG, which seems to make zero-shot transfer to new KGs, a key advantage of LLMs, difficult. It is also not verified whether the LLM’s general capabilities are retained after large-scale path-based fine-tuning, and the validity reward constrains outputs to the training KG rather than leveraging the open-ended expressiveness of a generative model. As it stands, “generative” refers to the output format rather than a demonstrated advantage. The authors should either add experiments supporting these claims or tone them down.

W2. The benefit of paths appears marginal relative to the overhead they introduce.

The paper argues that paths obtained by simple graph traversal can be spurious and that paths extracted from link predictors are therefore necessary. To this end, it incurs substantial overhead, including training link predictors, extracting paths, and constructing rewriting sets via set cover (data curation alone takes 9–11 hours per dataset). However, the benefit of paths is not clearly evident in the experiments. For example, removing path-augmented retrieval at test time reduces HITS@1 by only 0.5–0.6 points (Table 15), and CoT path supervision and the path-coverage reward also yield limited gains (Tables 6 and 7). Moreover, no experiment compares link-predictor paths against paths obtained by simple graph traversal to support the claimed advantage. A more thorough evaluation is needed to show whether the gains from link-predictor paths justify this substantial overhead.

Questions for authors

Q1. Section 4.2 states that PARR is evaluated zero-shot in the inductive experiments, yet the appendix separately lists link predictor hyperparameters for the inductive setting (Table 19) and the number of RL training samples for it (F.2). What data was PARR trained on for the inductive experiments? If it was trained on the target inductive splits, comparing it within the same zero-shot category as KG foundation models that are not trained on the target KG seems inappropriate. Conversely, if it was trained on the transductive KGs, leakage may occur, since the inductive splits are sampled from the original KGs and may overlap with the test data.

Q2. The efficiency comparison in Appendix C highlights that PARR is about 30× faster than KICGPT. However, KICGPT relies on a commercial LLM API, so its latency includes factors unrelated to the method itself, such as network latency and rate limits, which makes a direct comparison with a locally served model inappropriate. Could the authors provide efficiency comparisons with methods that use local LLMs under the same hardware setting?

5 comments
AuthorsOpenReview ↗

Reply (1/3)

W1: "The claimed benefits of the generative formulation are not demonstrated."

  • Regarding the meaning of "generative". "Generative" refers to the fact that PARR is trained and run purely through autoregressive generation. We use the term to contrast PARR with mainstream link predictors, which are trained with discriminative objectives (e.g., a BCE loss). Some LLM-based methods, such as KICGPT, KGR3 and DIFT, also generate autoregressively, but they rely on a pretrained link predictor to first produce a candidate set and use the LLM only to rerank it. This is a different, and much easier, task than generating the entities directly.

  • The advantage of the generative formulation. We have shown that PARR generalizes to unseen entities and relations, achieving strong zero-shot performance in both inductive settings (Tables 2 and 3) and surpasses the prior SOTA, ULTRA and TRIX, which are strong KG foundation models pretrained on more diverse KGs.

  • Scaling toward a foundation KGC model. We conduct an additional preliminary experiment, where we train PARR on the same three KGs used to pretrain ULTRA and TRIX (FB15k-237, WN18RR and CoDEx-M), with exactly the same training recipe as in the paper (denoted PARR-pretrain). Without any adaptation or hyperparameter tuning, PARR-pretrain outperforms ULTRA, TRIX and the PARR models reported in the paper on every dataset below. This confirms that PARR scales with more pretraining data.

HITS@10FB15k-237WN18RRYAGO3-10NELL-995WN18RR-ind v1FB15k-237-ind v1FB-100WK-100NL-100
Supervised SOTA (as reported by ULTRA)0.5990.6660.7080.6510.8260.5890.3710.1690.506
ULTRA0.5640.6140.6150.5430.7680.6560.6420.2860.651
TRIX0.5590.6110.6270.6290.7910.6820.6350.2990.676
PARR-RL-Qwen3 (reported in the paper)0.6390.7020.7730.7250.8660.7130.6880.3490.744
PARR-pretrain-RL-Qwen3 (same training KGs as ULTRA/TRIX)0.6630.7250.7840.7410.8770.7240.6900.3840.759

For ULTRA and TRIX, FB15k-237 and WN18RR are pretraining graphs and all other columns are zero-shot. We have added these results to the paper (Appendix D, Table 15).

"However, the experiments are limited to single-task link prediction on each KG separately, so they do not show whether the claimed strengths (e.g., generalizability and conversational use) actually materialize."

  • Generalizability. In the paper, generalizability refers to PARR's zero-shot performance in both inductive settings. The experiment above further shows that PARR transfers to new KGs and outperforms KG foundation models trained on the same data.
  • Multi-task and conversational use. We mention conversational and multi-task use as potential advantages of native LLM-based KGC, but they are outside the scope of this work, as noted in our Limitations (Appendix A). Making a native LLM strong at KGC is already a solid milestone on its own. A native LLM that answers any KG-related question in an open QA format is the subject of our follow-up work, which needs substantial further work on datasets, benchmarks and training.

"It is also not verified whether the LLM's general capabilities are retained after large-scale path-based fine-tuning."

Some loss of general capability (i.e. catastrophic forgetting) is expected for any LLM fine-tuned for a specialized task such as KGC; it is not specific to PARR: for instance, all the previous works that repurpose LLM as a discriminative link predictor (e.g. MKGL) completely sacrifice LLM's generation capability. On the other hand, as PARR is trained with LoRA, so the base model stays unchanged and the LoRA adapter can be detached for general use.

The validity reward constrains outputs to the training KG rather than leveraging the open-ended expressiveness of a generative model.

The validity reward only requires the generated paths to be valid. This matters for grounding paths: "open-ended expressiveness" in a grounding path amounts to hallucinated relations and shortcuts. The validity reward plays the same role as the format rewards widely used in RLVR, and we are not aware of evidence that this type of reward limits the expressiveness of a generative model.

AuthorsOpenReview ↗

Reply (2/3)

W2. The benefit of paths appears marginal relative to the overhead they introduce.

Table 6 shows the effect of SFT with and without paths. The rewriting stage trained with our Minimal Rewriting Sets (MRS) adds about 15–18 points of HITS@1 (FB15k-237: 0.155 → 0.302; WN18RR: 0.285 → 0.467), and CoT supervision with retrieval perturbation adds another 4–5 points (0.302 → 0.348 and 0.467 → 0.507). We have run two additional experiments:

  • RL applied directly, without SFT on the extracted paths;
  • SFT and RL with paths obtained by simple graph traversal.

We observe that:

  • RL without path-supervised SFT brings little improvement and can even hurt performance.
  • Using paths from simple graph traversal for retrieval and CoT can even degrade performance: both models fall below the no-CoT baseline. This is expected, since graph traversal has no notion of whether a traversed path supports the target relation.
ModelFB15k-237 H@1FB15k-237 H@3FB15k-237 H@10WN18RR H@1WN18RR H@3WN18RR H@10
PARR-SFT-Qwen3, no CoT0.3020.3880.4800.4670.4990.544
PARR-RL-Qwen3, no path-supervised SFT0.2850.3710.4930.4420.4900.538
PARR-RL-Qwen3, with path-supervised SFT0.3870.5240.6390.5640.6560.702
PARR-SFT-Qwen3, graph-traversal paths0.2410.3070.4250.2560.3080.373
PARR-RL-Qwen3, graph-traversal paths0.2550.3240.4110.2480.3160.397

For example, removing path-augmented retrieval at test time reduces HITS@1 by only 0.5–0.6 points (Table 15).

As shown above, paths are essential for training the rewriting and CoT stages. Table 15 of the submission (Table 17 in the revision) serves a different purpose: it shows that path-augmented retrieval is optional at test time, so the pretrained link predictors can be removed from inference. This is consistent with our main claim.

CoT path supervision and the path-coverage reward also yield limited gains (Tables 6 and 7).

  • CoT path supervision indeed yield significant gain, as shown above.
  • The path-coverage reward is an auxiliary term of the rewrite reward that gives dense feedback when none of the reasoning rollouts is correct (Sec. 3.6). Its modest gain is expected and is irrelevant of whether path supervision is needed for native LLM-based KGC.
AuthorsOpenReview ↗

Reply (3/3)

Q1. "Section 4.2 states that PARR is evaluated zero-shot in the inductive experiments, yet the appendix separately lists link predictor hyperparameters for the inductive setting (Table 19) and the number of RL training samples for it (F.2). What data was PARR trained on for the inductive experiments? If it was trained on the target inductive splits, comparing it within the same zero-shot category as KG foundation models that are not trained on the target KG seems inappropriate. Conversely, if it was trained on the transductive KGs, leakage may occur, since the inductive splits are sampled from the original KGs and may overlap with the test data."

The zero-shot results in Tables 2 and 3 are obtained with models trained only on the corresponding transductive training sets (e.g., FB15k-237 and WN18RR for the inductive splits in Table 2). PARR receives no training on any inductive split. The inductive hyperparameters in Table 19 of the submission (Table 25 in the revision) describe the link predictors' own inductive configurations, which PARR does not use. The 2.5K RL samples in Appendix F.2 refer to a fine-tuned inductive variant whose results are not included in the paper. We have removed this sentence from Appendix F.2.

Conversely, if it was trained on the transductive KGs, leakage may occur, since the inductive splits are sampled from the original KGs.

Training on the transductive KGs is the standard protocol for zero-shot inductive evaluation. ULTRA, TRIX, SEMMA and KRLM are all pretrained on the transductive training sets of FB15k-237, WN18RR and CoDEx-M, from which the GraIL inductive splits and FB-100 are sampled, so they share the same potential exposure and the comparison is fair. In addition, PARR-pretrain (see our reply to W1), trained on exactly the same three KGs as ULTRA and TRIX, outperforms them on every inductive benchmark in that table, including NL-100, whose source KG (NELL-995) is not part of this training set.

Q2: "The efficiency comparison in Appendix C highlights that PARR is about 30× faster than KICGPT. However, KICGPT relies on a commercial LLM API, so its latency includes factors unrelated to the method itself, such as network latency and rate limits, which makes a direct comparison with a locally served model inappropriate. Could the authors provide efficiency comparisons with methods that use local LLMs under the same hardware setting?"

ModelWN18RR sec/batchWN18RR max GPU mem. (GB)FB15k-237 sec/batchFB15k-237 max GPU mem. (GB)
MKGL3.157.54.759.3
KRLM3.760.45.564.1
KGR3 (re-implemented with vLLM)13.313510.3135
PARR7.4957.696

We report inference time on one H200 GPU with a batch size of 32 for three recent LLM-based KGC methods with strong performance, together with PARR. MKGL and KRLM repurpose the LLM as a discriminative predictor without autoregressive generation, which gives them higher throughput (about 1.4–2.4×). KGR3 follows a much more complicated inference protocol: it first obtains candidates from a pretrained link predictor, then performs in-context reasoning, and finally reranks. This requires several models to reside on the GPU: a link predictor such as NBFNet, an LLM for in-context reasoning, and a fine-tuned LLM for reranking. Reranking-based methods (KGR3, DIFT, DrKGC) also ask the LLM only for the top-1 entity and keep the remaining candidates in the link predictor's order. PARR performs query rewriting, retrieval and reasoning for the full top-10 list within a single LLM. This is much simpler, and is faster than KGR3 on both datasets (7.4 vs. 13.3 and 7.6 vs. 10.3 sec/batch).

Additional results from our replies to other reviewers.

We have also included additional experiments and ablations that could be potentially interesting to the reviewer. We therefore include a quick pointer below:

  • Biomedical KG. On PrimeKG, PARR outperforms the prior SOTA, KG-FIT, by 7.8 points of HITS@1 (0.683 vs. 0.605; reply to hck1, W3; Appendix D, Table 14). Together with the ULTRA-setting pretraining above, this shows that PARR's gains hold across domains and training regimes.
  • Validity of generated paths. RL with the validity reward raises the share of valid top-1 reasoning paths from 0.653 to 0.842 on FB15k-237 and from 0.739 to 0.907 on WN18RR (reply to qXKX, W3; Appendix E, Table 22), which supports our design of the validity reward discussed in W1.
  • Tree-structured rollouts. Crediting rewriting and reasoning separately adds 2.8 / 3.1 points of HITS@1 on FB15k-237 / WN18RR (reply to qXKX, W1; Appendix E, Table 20).
Reviewer_qXKXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PARR, a framework for training LLMs to do knowledge graph completion. The method extracts grounding paths from pretrained link prediction models, such as NBFNet and A*Net, and uses these paths to construct supervision for query rewriting and reasoning. The LLM is first trained by SFT, and then further optimized by GSPO with rewards for retrieval, prediction accuracy, and path validity. Experiments on multiple knowledge graphs show that PARR achieves strong performance in both transductive and inductive settings, and outperforms several traditional KGC models and LLM-based baselines.

Critical strengths
  1. The problem studied in this paper is meaningful. It tries to let general-purpose LLMs perform knowledge graph completion without modifying their architectures. Using structural information from pretrained link predictors is a reasonable idea.
  2. The experimental results are strong. PARR achieves large improvements over existing LLM-based methods, and also outperforms several strong traditional link prediction models. The evaluation covers multiple datasets and both transductive and inductive settings.
  3. The paper is generally well organized. The authors also provide useful ablation studies on retrieval, reasoning supervision, reward design, and noisy training paths.
Critical weaknesses
  1. The methodological contribution relative to existing teacher-distillation, retrieval, and RL pipelines is not sharply isolated. Among the most clearly new design elements are the construction of Minimal Rewriting Sets and the tree-structured GSPO training with KGC-specific rewards. Query rewriting, retrieval-augmented reasoning, path supervision, and GSPO are established techniques in general, although their combination for KGC is useful. The paper does not provide a controlled baseline with the same retriever, same teacher paths, same training data budget, and same RL budget. Such a baseline is needed to show why MRS and tree-structured GSPO are necessary beyond a simpler pipeline that directly distills teacher-generated paths into an LLM and then applies standard RL. Without this comparison, I am not fully convinced that the additional designs are sufficiently necessary as a methodological contribution.
  2. The evidence that PARR improves structural reasoning, rather than mainly improving retrieval, is not fully established. For example, Table 6 suggests that the trained Rewriter already brings a large gain, while the additional gain from CoT supervision is smaller on HITS@1 in some settings. This makes me doubt whether PARR mainly benefits from better retrieval, rather than improved structural reasoning. Moreover, PARR combines supervision from multiple pretrained link predictors, a large amount of constructed training data, and RL. Therefore, outperforming individual teacher models does not necessarily show that the LLM has learned stronger reasoning ability.
  3. The paper does not directly evaluate the faithfulness of generated reasoning paths. The validity rewards mainly check whether the paths follow the required format, contain existing graph edges, and avoid loops. But these checks do not necessarily prove that the paths logically support the target relation. Since path-supervised reasoning is central to the claim of this paper, the authors should provide a more direct evaluation of the correctness and faithfulness of the generated reasoning paths, instead of mainly relying on final link prediction accuracy.
Questions for authors
  1. What are the novelty and the essential significance of your proposal? I mean, beyond a standard teacher-distillation+SFT+RL pipeline, what is the essential technical contribution of MRS and tree-structured GSPO? A controlled comparison with a simpler distillation baseline under the same retriever, teacher paths, training data, and RL budget would be helpful.
  2. Since the validity rewards only check format, edge existence, and loops, please provide path-level evaluations.
3 comments
AuthorsOpenReview ↗

Reply (1/3)

W1: "The methodological contribution relative to existing teacher-distillation, retrieval, and RL pipelines is not sharply isolated... Without this comparison, I am not fully convinced that the additional designs are sufficiently necessary as a methodological contribution."

As stated in the abstract (L17) and introduction (L72–79), the main bottleneck we identify in adapting LLMs to KGC is the lack of path supervision, i.e., supervision for retrieval and reasoning. All previous LLM-based methods with strong performance either repurpose LLM as discriminative predictor (e.g. MKGL, KRLM), or use LLM to select top-1 from a candidate list obtained by using a pretrained link predictor. To solve this lang-standing issue, we propose to "leverage grounding paths extracted from pretrained interpretable link predictors as supervision for LLM-based KGC" (abstract, L17), which is our main methodological contribution.

"Query rewriting, retrieval-augmented reasoning, path supervision, and GSPO are established techniques in general, although their combination for KGC is useful."

We agree that query rewriting, retrieval-augmented reasoning and GSPO are established techniques in general. Our contribution is to make them work for KGC, where the supervision they need does not exist. Specifically:

  • We are the first to use grounding paths from pretrained link predictors as retrieval and CoT supervision for native LLM-based KGC.
  • We are the first to explore RLVR for KGC, training the full rewrite–retrieve–reason pipeline end-to-end with GSPO over two-level, tree-structured rollouts that credit rewriting and reasoning separately.
  • With these components, PARR achieves SOTA performance and is, to our knowledge, the first native LLM-based KGC method to surpass strong discriminative methods.

"Such a baseline is needed to show why MRS and tree-structured GSPO are necessary beyond a simpler pipeline that directly distills teacher-generated paths into an LLM and then applies standard RL."

Tree-structured GSPO. We ran an ablation without tree-structured rollouts, keeping all rewards the same (rewrite accuracy and coverage, NDCG@10 accuracy, format, validity). Tree-structured rollouts add 2.8 HITS@1 points on FB15k-237 and 3.1 on WN18RR:

ModelFB15k-237 H@1FB15k-237 H@3FB15k-237 H@10WN18RR H@1WN18RR H@3WN18RR H@10
PARR-RL-Qwen3, w/o tree-structured rollouts0.3590.4880.6040.5330.6150.683
PARR-RL-Qwen3, w/ tree-structured rollouts0.3870.5240.6390.5640.6560.702

We have added this ablation to the paper (Appendix E, Table 20).

MRS. MRS construction (Algorithm 1) is how we build the training supervision for KGC query rewriting, so removing it removes the rewriting stage altogether. Figure 3 and Table 6 already ablate this: with the Rewriter trained on MRS, retrieval recall is well above that of all non-learned retrievers with higher precision (Figure 3), and FB15k-237 HITS@1 improves from 0.155 to 0.302 (Table 6).

AuthorsOpenReview ↗

Reply (2/3)

W2: "The evidence that PARR improves structural reasoning, rather than mainly improving retrieval, is not fully established. For example, Table 6 suggests that the trained Rewriter already brings a large gain, while the additional gain from CoT supervision is smaller on HITS@1 in some settings. This makes me doubt whether PARR mainly benefits from better retrieval, rather than improved structural reasoning. Moreover, PARR combines supervision from multiple pretrained link predictors, a large amount of constructed training data, and RL. Therefore, outperforming individual teacher models does not necessarily show that the LLM has learned stronger reasoning ability."

"The evidence that PARR improves structural reasoning, rather than mainly improving retrieval, is not fully established. [...] This makes me doubt whether PARR mainly benefits from better retrieval, rather than improved structural reasoning."

  • Both retrieval and CoT supervision are essential parts of PARR. The query rewriting stage improves FB15k-237 HITS@1 from 0.155 to 0.302, and CoT supervision with RL raises it further to 0.387. Both gains are large.
  • A larger gain from retrieval than from CoT and RL does not contradict our claims. As stated in the abstract (L13–14) and introduction (L43–71), the bottleneck we identify is "the lack of retrieval and reasoning supervision in KGC datasets" (abstract, L13–14). We do not claim that PARR benefits mainly from structural reasoning; both components are needed.
  • Retrieval sets the base level of KGC performance; once the Rewriter is in place, further gains come from CoT supervision and RL (Tables 6 and 7).
  • We ran an additional ablation that skips path-supervised SFT and applies the same RL recipe directly. RL without path supervision is even below the no-CoT SFT model and far behind the runs trained with the extracted CoT (e.g., 0.285 vs. 0.387 HITS@1 on FB15k-237). This shows that grounding-path supervision is necessary in PARR.
ModelFB15k-237 H@1FB15k-237 H@3FB15k-237 H@10WN18RR H@1WN18RR H@3WN18RR H@10
PARR-SFT-Qwen3, no CoT0.3020.3880.4800.4670.4990.544
PARR-SFT-Qwen3, CoT (+ retr. perturbation)0.3480.4600.5860.5070.5810.644
PARR-RL-Qwen3, no path-supervised SFT0.2850.3710.4930.4420.4900.538
PARR-RL-Qwen3, with path-supervised SFT0.3870.5240.6390.5640.6560.702

"Moreover, PARR combines supervision from multiple pretrained link predictors, a large amount of constructed training data, and RL. Therefore, outperforming individual teacher models does not necessarily show that the LLM has learned stronger reasoning ability."

The two teacher link predictors, NBFNet and A*Net, are well below PARR on all benchmarks below, so PARR does more than reproduce the teachers' capability (Tables 1 and 4):

DatasetMetricNBFNetA*NetPARR-RL-Qwen3Gain over stronger teacher
FB15k-237HITS@10.3210.3210.387+6.6 (+20.6%)
HITS@30.4540.4530.524+7.0 (+15.4%)
HITS@100.5990.5860.639+4.0 (+6.7%)
WN18RRHITS@10.4970.4950.564+6.7 (+13.5%)
HITS@30.5730.5730.656+8.3 (+14.5%)
HITS@100.6660.6590.702+3.6 (+5.4%)
NELL995HITS@10.4850.4790.577+9.2 (+19.0%)
HITS@100.6550.6520.725+7.0 (+10.7%)
YAGO3-10HITS@10.4800.4700.598+11.8 (+24.6%)
HITS@30.6120.6110.707+9.5 (+15.5%)
HITS@100.7080.7070.773+6.5 (+9.2%)
ogbl-wikikg2MRR–0.6770.755+7.8 (+11.5%)

For scale, on FB15k-237 PARR's HITS@1 improvement over NBFNet (+6.6 points) is larger than NBFNet's own improvement over the previous GNN-based SOTA, CompGCN (0.264 → 0.321, +5.7 points). The same holds on WN18RR (+6.7 vs. +5.4 points; CompGCN 0.443 → NBFNet 0.497). Compared with our LLM baseline without path supervision (Tables 1 and 6), the gain is even larger: HITS@1 rises from 0.155 to 0.387 on FB15k-237 and from 0.285 to 0.564 on WN18RR. PARR thus greatly improves the LLM's own reasoning on KGC.

AuthorsOpenReview ↗

Reply (3/3)

W3: "The paper does not directly evaluate the faithfulness of generated reasoning paths. The validity rewards mainly check whether the paths follow the required format, contain existing graph edges, and avoid loops. But these checks do not necessarily prove that the paths logically support the target relation. Since path-supervised reasoning is central to the claim of this paper, the authors should provide a more direct evaluation of the correctness and faithfulness of the generated reasoning paths, instead of mainly relying on final link prediction accuracy."

Thank you for this comment. We evaluate the generated paths on two criteria.

(1) Whether the path itself is valid. This means the path is contiguous, has no loops and consists of existing edges and entities, which is what the validity reward checks. We additionally measured the percentage of valid top-1 reasoning paths:

ModelValid top-1 paths, FB15k-237Valid top-1 paths, WN18RR
PARR-SFT-Qwen30.6530.739
PARR-RL-Qwen3, w/o validity reward0.7260.854
PARR-RL-Qwen3, w/ validity reward0.8420.907

We observe that:

  • Most generated paths are valid, at a rate far above the corresponding HITS@1. For the final RL model, most errors therefore come from valid paths that do not support the target relation.
  • RL, and the validity reward raises path validity from 0.653 to 0.842 on FB15k-237 and from 0.739 to 0.907 on WN18RR.

(2) Whether the path logically supports the target relation. This is captured by the accuracy reward. A valid path ends at the predicted entity, so whether it supports the target relation is reflected directly in the accuracy reward and the final accuracy.

Q1: "What are the novelty and the essential significance of your proposal?"

Please see our reply to W1 for details. In short:

  • We identify the long-standing obstacle to native LLM-based KGC, the lack of supervision for retrieval and structural reasoning, and solve it with grounding paths extracted from pretrained link predictors.
  • We are the first to explore RLVR for KGC, with two-level, tree-structured rollouts.
  • PARR achieves superior performance across 19 KGs of different scales. Previous LLM-based KGC methods either repurpose the LLM as a discriminative predictor (e.g., MKGL, KRLM) or use it to rerank candidates produced by a pretrained link predictor (e.g., KICGPT, DIFT, KGR3, DrKGC); in DIFT, KGR3 and DrKGC, the LLM outputs only the top-1 entity, whereas PARR generates the full top-10 list. To our knowledge, PARR is the first native LLM-based KGC method to surpass strong discriminative methods.
  • PARR also achieves superior generalization. In both inductive settings (unseen entities, and unseen entities and relations), it surpasses the prior SOTA, KG foundation models designed for this purpose, by a clear margin.

Q2: "Since the validity rewards only check format, edge existence, and loops, please provide path-level evaluations."

Please see our reply to W3.

Additional results from our replies to other reviewers.

We have also included additional experiments and ablations that could be potentially interesting to the reviewer. We therefore include a quick pointer below:

  • Using paths from simple graph traversal (reply to Cz3R, W2; Appendix E, Table 21). Replacing link-predictor paths with paths from simple graph traversal lowers FB15k-237 HITS@1 from 0.387 to 0.255 after RL, below even the SFT model without CoT. This further demonstrates the necessity of PARR.
  • Pretraining in the ULTRA setting (reply to Cz3R, W1; Appendix D, Table 15). To test the scalability and generalizability of PARR towards a KG foundational model, we trained on the same three KGs as ULTRA and TRIX (FB15k-237, WN18RR, CoDEx-M). PARR outperforms both on all nine transductive and inductive benchmarks we test, e.g., 0.759 vs. 0.676 HITS@10 on NL-100.
  • Biomedical KG (reply to hck1, W3; Appendix D, Table 14). We conduct additional experiments on KG dataset that is more out-of-domain of LLM's pretrained knowledge. On PrimeKG, PARR outperforms the prior SOTA, KG-FIT, by 7.8 points of HITS@1 (0.683 vs. 0.605).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.