- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether grounded KGQA systems provide sufficient evidence for their answers, rather than simply producing the correct answer. It proposes an evaluation protocol based on query constraints and support-removal interventions, and evaluates it on WebQSP, CWQ, and 2WikiMultiHopQA. The paper also introduces GRASP to improve evidence grounding and abstention.
Critical strengths
-
The paper is easy to understand. The main concepts, including evidence sufficiency, binding consistency, and support removal, are explained with clear examples.
-
The motivation is clear. The paper makes a reasonable case that answer accuracy alone is not enough when a system is expected to answer only from a given knowledge source.
-
The evaluation design is concrete. Query-derived constraints and controlled graph edits provide a direct way to test evidence sufficiency and behavior after support is removed.
Critical weaknesses
-
The main observation is close to findings already reported in Sufficient Context (Joren et al., 2025), FaithEval (Ming et al., 2025), and When Silence Is Golden (Zhou et al., 2026). These works have already shown that LLMs may answer when the available evidence is insufficient. The paper should clarify more clearly what new insight is obtained from studying this problem in the KGQA setting.
-
The design of EVIDENCEGAP is not well justified. It is defined as 1 − E-HIT/HIT@1, but does not account for cases where the model answers from its own parametric knowledge. In other words, a correct answer may still be counted as an evidence gap simply because the relevant fact is not present in the returned evidence. This is especially important for general-domain datasets such as CWQ and WebQSP, since models such as Qwen2.5-7B, Llama-3.1-8B, and GPT-6 may already have learned some of the facts in these datasets. The authors should discuss the evidence boundary of parametric knowledge and provide suitable controls.
-
The experimental setup is not fully clear, and the fairness of the comparisons is difficult to assess. For example, Table 4 does not provide enough information about the Codex/GPT-6 configuration or prompts. It is also unclear why RoG, VoG, and SubgraphRAG are not included as baselines, although they are discussed in Table 1. In addition, most open-source experiments use 7–9B models. Including models around 13B, as used by methods such as RoG and GCR, would make the comparison more balanced. Given these limitations, the current evidence is not sufficient to fully support the conclusions of RQ1.
-
The support criterion relies heavily on gold SPARQL queries or annotated evidence. This may judge semantically valid evidence as insufficient when it uses a different schema or relation path. The semantic-relaxation analysis is helpful, but may not fully resolve this issue.
Questions for authors
See above.