- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies foundation retrievers for multi-hop question answering over knowledge graphs. Its goal is to train a retriever on one or more source KGs and transfer it to a different KG, where relation labels, graph structure, and question distributions may change. The authors analyze a prior KG foundation retriever and report that correct answers often remain reachable after transfer but receive poor ranks. They propose KGSeek, which combines question–relation semantic relevance with graph-neighborhood structure, aggregates relevance along paths, and applies bounded, length-normalized contextual adjustments. Joint training over source KGs is intended to produce scoring functions that transfer through relation labels and graph structure. Experiments on Freebase and DBpedia report improvements in cross-KG retrieval and downstream QA, as well as competitive or improved in-domain results with different LLM readers. The main unresolved issue for me is whether this transfer setting has a compelling practical role relative to adapting or fine-tuning a retriever on the target KG using target-graph training data.
Critical strengths
-
The paper identifies a concrete ranking failure mode in cross-KG transfer: correct answers may remain reachable but receive low ranks because the retriever does not transfer its source-KG preferences effectively. Combining semantic relevance with graph structure is a reasonable response to that failure mode.
-
The evaluation appears to consider both retrieval and downstream QA, and it reports results across Freebase and DBpedia with multiple LLM readers. This is useful evidence that the proposed scoring architecture is not limited to a single reader or a single in-domain benchmark.
Critical weaknesses
-
The practical use case for zero-shot or source-trained transfer is not established. In a practical KG-based QA system, the retriever is often adapted or fine-tuned to the target graph when target-graph questions, answers, or retrieval supervision are available. Such adaptation may be preferable because it can directly exploit the target graph’s schema, relation semantics, and question distribution. I do not assume that target supervision is always available: transfer could be valuable for newly created KGs, low-resource domains, or systems that must support many graphs. However, the paper does not identify and validate such a deployment regime, nor does it compare against target-KG adaptation and the associated quality/compute/data trade-offs.
-
The paper’s main contribution may improve a benchmark transfer setting without showing a consequential downstream benefit. Cross-KG retrieval is an interesting scientific problem, but the experiments do not demonstrate that KGSeek reduces deployment cost, eliminates target-KG fine-tuning, improves QA in a realistic low-resource regime, or enables a use case that target-specific training cannot support. The reported cross-KG gains therefore do not yet establish significance for a broad conference. A stronger paper would show when transfer is preferable to adaptation, how much target data it saves, and whether its advantage persists under matched compute and retrieval budgets.
Questions for authors
-
Who is the intended user or deployment scenario for a source-trained retriever on a new KG? Please describe a realistic setting in which target-KG questions/answers or retrieval labels are unavailable, delayed, or too expensive to obtain, and explain why KGSeek is preferable to an unsupervised retriever in that setting.
-
How does KGSeek compare with fine-tuning or otherwise adapting the same retriever on the target KG when a modest amount of target-graph supervision is available? Please report quality-versus-target-data and quality-versus-compute trade-offs, including the point at which target adaptation overtakes zero-shot transfer. This comparison is necessary to establish when the proposed transfer setting is practically preferable rather than merely possible.
-
Do the cross-KG gains remain when the target KG has relation labels or graph schemas that are not aligned with the source KGs? The method appears to rely on relation labels and shared scoring functions; please clarify what information is assumed to be shared or available at transfer time.
-
Are the downstream QA gains driven by improved retrieval itself, or by differences in retrieved context size, answer candidate set, or LLM reader? Please include matched-budget comparisons and report whether the gain persists against a target-specific baseline.
-
What practical resource does the proposed method save—target-graph annotation, retriever training, inference cost, or engineering effort—and how is that saving measured? Without this analysis, it is difficult to assess the significance of the transfer objective.