Grounding Is Not Enough: Testing Evidence Sufficiency in LLM Reasoning over Knowledge Graphs

Grounded large language model (LLM) systems are expected to justify predictions with evidence, making answer correctness alone insufficient. Knowledge graph question answering (KGQA) provides a concrete setting for testing this requirement since returned graph evidence can be checked against the conditions needed to support a prediction. Yet existing KGQA evaluations largely emphasize answer correctness, leaving two questions insufficiently examined: whether the returned evidence sufficiently supports the prediction, and whether systems return Unknown when the available knowledge cannot support an answer. We introduce an executable evaluation protocol that derives constraints from structured queries, verifies whether the returned evidence satisfies those constraints, and uses controlled graph edits to test system behavior after answer-related evidence is removed. We apply this protocol on WebQSP, CWQ, and 2WikiMultiHopQA. Across existing grounded LLM-based KGQA systems, the required supporting evidence is unavailable in up to 76.77% of cases, while systems correctly return Unknown in only 0.00–30.53% of cases where support is unavailable. After the removal of answer-critical evidence, 53.98% of initially correct predictions still persist, whereas only 18.31% transition to Unknown. These gaps persist even with Codex (GPT-6), suggesting that stronger language models alone do not eliminate the problem. Motivated by these findings, we introduce Grounding with Requirement-Aware Shared-binding Preservation (Grasp) as a reference mitigation that preserves question constraints, reducing the EvidenceGap to 27.47%, improving the correct Unknown up to 88.01%. Overall, our results reveal a key limitation of answer-based evaluation for grounded LLMs: it does not capture whether returned evidence sufficiently supports an answer or whether models abstain when such support is unavailable.