acceptodds
Under review as a conference paper at ICLR 2027

WHEN IS GRAPH EVIDENCE ENOUGH FOR FINE- GRAINED DISCRIMINATION IN INDUCTIVE KGC?

Abstract

Knowledge graph foundation models (KGFMs) can reject implausible entities yet fail to distinguish the correct answer from relation-compatible alternatives. For both ULTRA and FLOCK, replacing random candidate pools with relation-compatible pools of matched pre-filter size reduces mean reciprocal rank by 0.238–0.261, exposing a fine-grained discrimination failure. We investigate candidate-conditioned evidence retrieval as a remedy: a KGFM proposes candidates, and an LLM verifies each hypothesized relation against externally acquired support. Holding the shortlists and verifier model fixed, we compare entity names, bounded graph context, and Web evidence, and observe two contrasting evidence regimes. On FB15k-237Ind v1, Web verification improves conditional mean reciprocal rank over graph verification by 0.077–0.154 and also outperforms the original KGFM ranking. On pooled NELL-995Ind v4 samples, graph context improves over names, whereas Web evidence yields no statistically reliable further gain. Retrieval analysis associates failures with missing answer support or competing evidence: evidence availability alone does not ensure discrimination. These findings demonstrate the conditional value of candidate-level evidence acquisition as a complement to structural ranking and motivate selective retrieval for resolving plausible alternatives under the evaluated protocols.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
inductive knowledge graph completion, knowledge graph foundation models, knowledge graph reasoning, candidate verification, external knowledge

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DzZVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how external evidence affects the discrimination of relation-compatible candidates (i.e., candidates entities match the relations's expected entity type) in inductive knowledge graph completion. The authors first show that existing knowledge graph foundation models (ULTRA and FLOCK) perform substantially worse when ranking such candidates than random candidates. To address this issue, they propose a verification pipeline in which a KGFM generates a shortlist of candidates, and an LLM reranks them using different sources of information, including entity names, local graph context, and external Web evidence. Experiments on two datasets show that Web evidence improves candidate ranking on FB15k-237Ind-v1, while providing no significant additional benefit over graph evidence on NELL-995Ind-v4. The authors further analyze evidence availability and competing support to investigate the differences between these results.

Critical strengths
  • The paper identifies a concrete failure mode of KGFMs.
  • The analysis of answer support and competing support provides an explanation for why Web evidence helps on FB15k-237Ind-v1 but not consistently on NELL-995Ind-v4.
Critical weaknesses
  • The experimental evaluation is limited to only two subsets of commonly used inductive KGC benchmarks, which is insufficient to establish a consistent trend regarding when external evidence is beneficial.
  • The paper repeatedly mentioned selective retrieval, but does not actually evaluate whether the need for retrieval can be predicted at the query level. The current results show only that external evidence helps in some settings, without providing a practical way to decide when it should be used. It would be interesting to develop an adaptive strategy that selectively retrieves external evidence when needed.
Questions for authors
  • Have the authors tested whether the observed benefit of external evidence holds on additional inductive KGC datasets beyond FB15k-237Ind-v1 and NELL-995Ind-v4?
  • Can the authors clarify how sensitive the results are to the choice of verifier model? Would the same evidence regimes be observed with a different LLM?
Reviewer_Ap41OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when external evidence benefits inductive KGC. The authors identify a candidate-discrimination problem in existing KGFMs and propose a framework that uses frozen KGFMs to generate candidate shortlists, followed by LLM-based verification using entity names, graph context, or Web evidence. Experiments with ULTRA and FLOCK on two benchmarks show that Web evidence improves candidate ranking on FB15k-237Ind v1, while graph evidence is more beneficial on NELL-995Ind v4.

Critical strengths
  1. The paper highlights a meaningful limitation of existing KGFMs, where strong performance against randomly sampled negatives does not necessarily translate into effective discrimination among relation-compatible candidates. The comparison between random and relation-compatible candidate pools provides evidence of this problem and motivates more fine-grained evaluation of KGC models.

  2. The authors evaluate two KGFM backbones, ULTRA and FLOCK, under different evidence conditions. The paper also provides paired bootstrap confidence intervals and several sensitivity analyses, including investigations of evidence availability, competing support, and tie-breaking strategies. These experiments offer useful insights into the potential benefits and limitations of external evidence for candidate verification.

Critical weaknesses
  1. The methodological novelty is limited. The proposed framework primarily combines existing components, including pretrained KGFMs for candidate generation, Web search for evidence acquisition, predefined evidence categorization, and LLM-based reranking. The paper does not introduce a novel model architecture, learning objective, reasoning algorithm, or principled evidence selection mechanism. Similar retrieval-augmented KGC frameworks, such as KGR3 and KICGPT, have already explored combining retrieval, contextual information, and LLM-based reasoning. The paper does not sufficiently establish a substantial methodological advancement over these approaches.

  2. The construction of relation-compatible candidate pools raises concerns about evaluation validity and practical applicability. Specifically, the candidate pools are constructed using both observed graph edges and held-out evaluation triples, meaning that information from the evaluation set contributes to determining candidate eligibility.

  3. The paper does not adequately answer its central research question of when external evidence is beneficial. The main finding is that Web evidence improves performance on one benchmark but provides no statistically reliable additional benefit on another. However, differences between two datasets are insufficient to systematically characterize the conditions under which external evidence helps.

  4. The Graph-Only baseline is relatively weak, and the comparisons among evidence conditions are insufficiently controlled. Graph evidence extraction is restricted to a small number of one-hop facts and short connecting paths, selected through deterministic procedures rather than query-aware relevance estimation. This restricted representation may omit important structural information, making it difficult to conclude that Web evidence is inherently more informative than graph evidence.

  5. The experimental evaluation lacks sufficient breadth and systematic ablation studies. The main Web verification experiments are limited to two specific benchmark splits, two KGFM backbones, and a single LLM verifier, making it difficult to assess generalizability. Important factors, such as candidate shortlist size, graph evidence budget, retrieval strategy, LLM capability, and evidence selection criteria, are not systematically investigated.

Questions for authors

Address the problems in Weaknesses

Reviewer_N1SQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates when external evidence improves candidate ranking in inductive knowledge graph completion. It compares random and relation-compatible candidate pools, then uses a frozen LLM to rerank ULTRA and FLOCK shortlists under names-only, local graph evidence, and Web evidence conditions. Experiments show that relation-compatible candidates are harder to distinguish. Conditional on the correct answer being shortlisted, Web evidence improves ranking on FB15k-237Ind but provides no reliable additional benefit over graph evidence on NELL-995Ind. Further analysis examines missing answer support and competing evidence as possible explanations for these differences.

Critical strengths

None.

Critical weaknesses
  1. Implementation issues weaken the conclusions. Specifically:
  • Some NELL/FLOCK Names-only prompts display “known entity → candidate” even for head prediction, creating conflicting relation directions (p. 15, lines 791–797).
  • Identifier-derived entity names and relation descriptions may be ambiguous, without their effects being isolated (p. 16, lines 833–837).
  • Names permits general knowledge, whereas Graph/Web restrict it; Names/Graph use 0–10 scores, while Web uses 0–100.

The authors should correct the direction errors and rerun comparisons with controlled input wording, knowledge-use instructions, and scoring scales to clarify which gains come from the evidence.

  1. Limited explanatory insight. The paper reports gains on FB but an unresolved Web-over-Graph benefit on NELL, without adequately explaining this difference. The analysis mainly associates performance with missing answer support and competing evidence, but does not isolate their effects or establish a deeper mechanism. Consequently, the findings offer limited insight beyond the observation that retrieval sometimes helps, and provide little guidance on when these benefits should generalize.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.