- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies bias in string-based evaluation for generative knowledge-graph question answering. The authors observe that verbose responses may accidentally contain the correct answer or one of its aliases and therefore receive credit despite being semantically incorrect. The paper proposes the Analytic Debiasing Evaluator (ADE). ADE first performs alias-aware string matching. For each question, it then selects incorrect answers from the benchmark answer inventory that are similar to the gold answer in type, alias length, frequency, and graph-neighborhood properties. These answers serve as negative controls. If a response matches the gold answer but also matches multiple negative controls, ADE reduces its reliability score.
Critical strengths
- The paper identifies a real problem: the presence of the gold-answer string does not necessarily mean that the model selected it as its answer.
- ADE is deterministic, inexpensive, reproducible, and does not depend on a potentially unstable LLM judge.
Critical weaknesses
- Negative-control collisions do not identify the semantic role of the gold answer. Suppose the gold answer is Paris, while the response states, “Paris is a candidate, but the final answer is Lyon.” Unless Lyon happens to be included in the sampled negative-control set, ADE may still assign a high score. Conversely, a semantically correct response may be penalized for mentioning related entities that happen to appear among the negative controls. ADE therefore primarily measures whether a response mentions multiple entities, not whether the gold answer is selected, rejected, or merely considered as a candidate.
- ADE remains a heuristic discount applied to string matching rather than an evaluator of semantic correctness. It can only reduce the scores of responses that already match the gold answer. It cannot reliably handle paraphrases, negation, multi-entity answers, or correct responses that do not use a known alias. Its theoretical interpretation also depends on an exchangeability assumption between the gold answer and the matched negative controls. Matching entities by type, alias length, frequency, and graph statistics is insufficient to establish that they have the same probability of being mentioned in a particular response.
Questions for authors
- If a response mentions the gold answer but explicitly selects another answer that is not included in the negative-control set, how does ADE avoid assigning a high score?
- Can the authors provide evidence that the matched negative controls and the gold answer have exchangeable lexical-collision risks within individual responses?
- How stable are per-response ADE scores and model rankings under different random samples of negative controls?