- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces relation-centric knowledge graph question answering, where the goal is to identify informative relational subgraphs connecting given entities rather than retrieving a single answer entity. The proposed UniRel framework combines multi-stage subgraph retrieval with reinforcement learning-based LLM reasoning, using rewards that encourage connectivity, compactness, and informativeness. Experiments across seven knowledge graphs demonstrate improvements over existing baselines, supported by ablation studies, cross-domain evaluations, and human assessments.
Critical strengths
(1) The paper introduces a meaningful relation-centric KGQA setting that complements traditional entity-centric question answering and addresses a practically relevant problem. (2) The proposed framework effectively combines structured subgraph pruning with reward-guided LLM reasoning, providing a coherent approach to generating informative relational answers. (3) The experiments cover seven knowledge graphs and multiple LLM backbones, with strong results across baseline comparisons, ablation studies, and cross-domain generalization.
Critical weaknesses
(1) The evaluation relies heavily on metrics derived from the same reward function used for training, making it difficult to independently assess whether the generated subgraphs are genuinely more informative. (2) The relation-centric queries are primarily constructed from sampled entity pairs and manually designed templates, which may not adequately reflect the complexity and diversity of real-world user questions. (3) The human evaluation covers only four queries, limiting the strength of the evidence that UniRel's preference-based objective aligns with human judgments across different domains and query types. (4) The pipeline figure is messy.
Questions for authors
Could the authors evaluate UniRel on a broader set of naturally occurring relation-centric questions to demonstrate that its improvements generalize beyond template-generated queries?
4 comments
Response to W3 and W4
We have extended the human study from 4 to 35 queries covering all seven KGs, with 205 participants. UniRel is preferred over the baseline in 61–64% of judgments and over each reward ablation in 55–58%, with all comparisons significant.
(1) Setup.
- Queries. We selected five queries per KG, for which GNN-RAG and SubgraphRAG return a valid (connected) answer. On these queries, the two baselines return the same answer. As in the original study, participants compared four answers: UniRel, the baseline, w/o and w/o .
- Design. Each query asks participants to read four answers and rank them on three criteria, which is demanding. Longer questionnaires are known to lower response quality, as participants answer faster and less carefully toward the end [1]. We therefore kept each session to five queries, which took a median of 13 minutes, and used a between-subjects design to cover all 35 queries: participants were split into seven groups of five queries each, and each group covers five different KGs. Participants ranked the four answers by informativeness, coherence and usefulness, without knowing which method produced which answer.
- Participants. We recruited participants on Prolific. After pre-specified exclusions (e.g., failed attention check, completion under 5 minutes), 205 participants remained, 28–30 per group.
(2) Results. We report statistics in the same format as Appendix C.
| vs. baseline | vs. w/o | vs. w/o | |
|---|---|---|---|
| Informativeness | 64% | 55% | 57% |
| Coherence | 61% | 56% | 58% |
| Usefulness | 64% | 56% | 56% |
- Pairwise preference. The table above reports how often participants rank UniRel above each alternative. Each cell is based on about 1,000 judgments, and all nine comparisons are significant (, with standard errors clustered by participant).
- First place. UniRel is ranked first in 34% of informativeness judgments, against 29%, 21% and 16% for w/o , w/o and the baseline.
- Agreement. Kendall's is significant (Friedman test, ) on 29 of 35 queries for informativeness and usefulness, and on 23 for coherence. As in the original study, participants agree least on coherence.
(3) Participants' own descriptions. As in the original study, participants described what guided their choices in a free-text question. Many explicitly said they preferred specific connections over generic or broad ones. Representative responses include:
- "I just picked the ones with real, specific connections and put the generic ones like gender or nationality at the bottom."
- "I ranked answers higher when they linked entities through concrete facts (like shared careers or historical events) and ranked generic links (like gender or broad locations) at the bottom."
- "I favored granular entities for Informativeness."
As before, the instructions never mentioned node degree, relation frequency or compactness, so this preference was not suggested to participants.
(4) Variation across KGs. UniRel is preferred over the baseline on five of the seven KGs, most strongly on FB13 (88% on informativeness), DBpedia50 (72%), FB15k (71%) and YAGO3-10 (69%). It is not preferred on DBpedia500 and UMLS. Our response to Reviewer paDH (Q3) analyzes these cases, which point to two failure modes of our informativeness measures: rare but meaningless relations, and abstract ontologies where node degree carries little signal.
We will add the full study, including per-KG and per-query results, to Appendix C.
W4: Thank you. Fig. 1 has two parts. Fig.1(a) contrasts entity-centric and relation-centric KGQA. Fig.1(b) shows the two main components of UniRel: subgraph retrieval and RL-tuned LLM reasoning. We kept the multi-stage pruning out of this overview and show it separately in Fig. 2, so that the overview stays readable. In the revised version, we will expand the caption of Fig. 1 to explain how to read each part.
References:
[1] Galesic and Bosnjak, "Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey", Public Opinion Quarterly 2009.
Thank you for the detailed rebuttal and the extended evaluation, which address several of my concerns. However, I remain concerned about the limited diversity of naturally occurring queries and the inconsistent preference results across datasets.
Response to remaining concerns
Thank you for the follow-up. We respond to the two remaining points below.
1. Diversity of naturally occurring queries.
- Question types in entity-centric KGQA. Entity-centric KGQA asks for a specific target, and the question word follows from the type of that target. In MetaQA, for example, who questions ask for a person (e.g., "Who directed [movie]?"), when questions for a year (e.g., "When was [movie] released?"), what questions for an attribute (e.g., "What genre is [movie]?"), and which questions for a set of movies (e.g., "Which movies did [actor] star in?").
- Relation-centric questions are common in practice. Many real-world questions aim to understand how entities are associated rather than which entity satisfies a query, for example in education, scientific exploration and decision support. Here the target is the connection itself, and no target relation or answer type is given, so the who/when/what distinction does not apply. The diversity instead lies in the kind of connection the model must discover, such as a shared director, a shared genre or a family tie.
- Our queries inherit the variety of entity-centric questions. Most of our queries are converted from entity-centric questions of all these types. A who question gives a (movie, person) pair, a which question a (person, movie) pair, and a multi-hop question gives a pair that is several hops apart. Beyond two-entity queries, we also construct queries with three and four entities.
- We follow MetaQA's construction practice. MetaQA generates its questions from templates: each question type, defined by its reasoning path, has 10 text templates. We likewise generate relation-centric questions from 8 templates. The difference is that MetaQA fixes the reasoning path for each question type, while our task leaves the path open for the model to discover.
- No existing benchmark. To our knowledge, no existing benchmark contains naturally occurring relation-centric questions. If the reviewer has suggestions for better ways to construct such a dataset, we would be glad to hear them and to include experiments on it in the revised version.
2. Consistency across datasets.
Note that to compare answer quality rather than validity, the human study only includes queries where the baseline also returns a valid (connected) answer.
- Clarification of our earlier statement. In our previous response, we wrote that UniRel “is not preferred” on DBpedia500 and UMLS. To be precise, on these two KGs UniRel is on par with the baseline in informativeness (50% and 53%, not significantly different), rather than worse. Overall, participants prefer UniRel over the baseline on every criterion ().
- The remaining gap reflects properties of the KGs. This matches the limitation cases we analyzed in our response to Reviewer paDH (Q3). DBpedia contains schema-level relations, such as type_of_union, that are rare but carry little meaning. UMLS is a biomedical ontology. Its nodes are abstract semantic types, such as Family Group or Fungus, rather than concrete entities like people or films. Every answer is therefore a chain of categories, and node degree reflects how general a type is in the ontology, not how informative an answer is to a reader. In both KGs, the proxies behind and carry less signal than in KGs of real-world entities, so optimizing them helps less. Since UniRel builds its answers from KG facts, answer quality depends on KG quality, as noted in Sec. 7.
As the first framework for the relation-centric KGQA task introduced in this paper, UniRel already improves human-judged informativeness with simple proxies. As better proxies are developed, for example personalized ones (Sec. 7), they can be used directly within the same framework.
We hope this addresses your remaining concerns.
Response to W1 and W2
We thank the reviewer for the feedback, and apologize for the late response. We used the time to run an extended human evaluation (35 queries across all seven KGs, 205 participants), which directly addresses W3 and supports W1.
W1: We agree with this concern, and the paper states it directly at the start of Sec. 5: ``Our automatic metrics all rely on the reward we optimize.'' This is why we ran an independent human evaluation.
So, without being told what our reward favors, people judge the answers it produces as more informative and useful.
W2: Our queries are grounded in existing KGQA benchmarks, and the diversity of the task lies in the relational structures the model must discover rather than in the wording of the question.
Please refer to our response to Reviewer paDH (Q1) for the detailed construction procedure and per-KG sources, and to our response to Reviewer eB42 (Q2) for the full template list.