- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how external evidence affects the discrimination of relation-compatible candidates (i.e., candidates entities match the relations's expected entity type) in inductive knowledge graph completion. The authors first show that existing knowledge graph foundation models (ULTRA and FLOCK) perform substantially worse when ranking such candidates than random candidates. To address this issue, they propose a verification pipeline in which a KGFM generates a shortlist of candidates, and an LLM reranks them using different sources of information, including entity names, local graph context, and external Web evidence. Experiments on two datasets show that Web evidence improves candidate ranking on FB15k-237Ind-v1, while providing no significant additional benefit over graph evidence on NELL-995Ind-v4. The authors further analyze evidence availability and competing support to investigate the differences between these results.
Critical strengths
- The paper identifies a concrete failure mode of KGFMs.
- The analysis of answer support and competing support provides an explanation for why Web evidence helps on FB15k-237Ind-v1 but not consistently on NELL-995Ind-v4.
Critical weaknesses
- The experimental evaluation is limited to only two subsets of commonly used inductive KGC benchmarks, which is insufficient to establish a consistent trend regarding when external evidence is beneficial.
- The paper repeatedly mentioned selective retrieval, but does not actually evaluate whether the need for retrieval can be predicted at the query level. The current results show only that external evidence helps in some settings, without providing a practical way to decide when it should be used. It would be interesting to develop an adaptive strategy that selectively retrieves external evidence when needed.
Questions for authors
- Have the authors tested whether the observed benefit of external evidence holds on additional inductive KGC datasets beyond FB15k-237Ind-v1 and NELL-995Ind-v4?
- Can the authors clarify how sensitive the results are to the choice of verifier model? Would the same evidence regimes be observed with a different LLM?