acceptodds
Under review as a conference paper at ICLR 2027

When Verbose Responses Mislead Evaluators: Analytic Debiasing for Generative KGQA

Abstract

Knowledge Graph Question Answering (KGQA) aims to answer natural-language questions over a knowledge graph. Classical KGQA systems typically return a single canonical answer entity or an executable logical form, so exact-match evaluation is usually well aligned with their outputs. Recent systems, however, increasingly generate free-form natural-language responses. This makes string matching less reliable: it can reject correct answers with mismatched surface forms and accept incorrect responses that merely mention answer-like entities. We formalize this issue as evaluator bias in generative KGQA. In our framework, observed scores reflect not only latent semantic correctness, but also false-negative mismatch and false-positive contamination. We then propose the Analytic Debiasing Evaluator (ADE), a simple, interpretable, and judge-free framework. ADE starts from matches accepted by a string-based evaluator and assigns each accepted match a reliability score based on its risk of lexical contamination. ADE estimates lexical-contamination risk using matched negative controls and discounts accepted matches that are likely to be contamination-prone. Experiments on human-annotated meta-evaluation sets show that ADE aligns more closely with human judgments than string-matching evaluators. ADE also achieves competitive agreement with LLM-as-Judge evaluators while being deterministic, faster, and free of API cost. In leaderboard analyses, ADE produces contamination-aware rankings that better match human semantic judgments, suggesting that it can help diagnose when biased evaluators distort model rankings.

Primary area:
datasets and benchmarks
Keywords:
Knowledge Graph Question Answering, LLM Evaluation, Automatic Evaluation, Evaluator Bias, Lexical Contamination, Negative Controls

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ckm2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies bias in string-based evaluation for generative knowledge-graph question answering. The authors observe that verbose responses may accidentally contain the correct answer or one of its aliases and therefore receive credit despite being semantically incorrect. The paper proposes the Analytic Debiasing Evaluator (ADE). ADE first performs alias-aware string matching. For each question, it then selects incorrect answers from the benchmark answer inventory that are similar to the gold answer in type, alias length, frequency, and graph-neighborhood properties. These answers serve as negative controls. If a response matches the gold answer but also matches multiple negative controls, ADE reduces its reliability score.

Critical strengths
  1. The paper identifies a real problem: the presence of the gold-answer string does not necessarily mean that the model selected it as its answer.
  2. ADE is deterministic, inexpensive, reproducible, and does not depend on a potentially unstable LLM judge.
Critical weaknesses
  1. Negative-control collisions do not identify the semantic role of the gold answer. Suppose the gold answer is Paris, while the response states, “Paris is a candidate, but the final answer is Lyon.” Unless Lyon happens to be included in the sampled negative-control set, ADE may still assign a high score. Conversely, a semantically correct response may be penalized for mentioning related entities that happen to appear among the negative controls. ADE therefore primarily measures whether a response mentions multiple entities, not whether the gold answer is selected, rejected, or merely considered as a candidate.
  2. ADE remains a heuristic discount applied to string matching rather than an evaluator of semantic correctness. It can only reduce the scores of responses that already match the gold answer. It cannot reliably handle paraphrases, negation, multi-entity answers, or correct responses that do not use a known alias. Its theoretical interpretation also depends on an exchangeability assumption between the gold answer and the matched negative controls. Matching entities by type, alias length, frequency, and graph statistics is insufficient to establish that they have the same probability of being mentioned in a particular response.
Questions for authors
  1. If a response mentions the gold answer but explicitly selects another answer that is not included in the negative-control set, how does ADE avoid assigning a high score?
  2. Can the authors provide evidence that the matched negative controls and the gold answer have exchangeable lexical-collision risks within individual responses?
  3. How stable are per-response ADE scores and model rankings under different random samples of negative controls?
Reviewer_FHEnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes errors in the evaluation of generative knowledge graph question-answering (KGQA) systems. A response might be deemed correct simply because it contains the reference answer, even when it does not actually provide that answer. The authors propose ADE, a method that first employs a string-matching approach capable of recognizing aliases, and subsequently reduces the score assigned to responses deemed correct if they also coincide with predefined incorrect answers. The experiments compare ADE with other evaluation systems—based on both string matching and the use of LLMs—using human annotations, and examine how the choices made during the evaluation phase influence the ranking of the models.

Critical strengths

1) The method could serve as a practical evaluation tool that does not require an additional LLM to act as a judge.

In fact, ADE does not need to call upon models to assign scores to responses. This can be useful when a large number of responses must be evaluated quickly, while also keeping computational costs in check.

2) The evaluation incorporates human annotations.

Each response is examined independently by three annotators. Comparing the decisions made by ADE with the judgments provided by these annotators offers an effective way to verify whether the proposed correction actually improves the evaluation.

Critical weaknesses

1) The practical scope and utility of the method require a clearer justification.

I understand that permissive string matching may accept incorrect answers. However, this is a general issue in the evaluation of QA systems, not one specific to KGQA. It is not clear to me how essential the KG is to the proposed solution: features such as the length or frequency of aliases do not require a graph, unlike neighborhood matching. Furthermore, the implementation includes a fallback mechanism for cases where neighborhood information is unavailable. I would therefore appreciate it if the authors could clarify which resources are strictly necessary for ADE and to what extent KG-specific information contributes to its performance.

This point is crucial for understanding the contexts in which the method can realistically be applied. In KGQA, the graph is already available. In other QA scenarios, however, constructing a graph or compiling an inventory of entities—along with the required metadata—solely for evaluation purposes could be burdensome. I am not asking the authors to test every possible type of QA task, but rather to clarify the intended scope of application and the requirements for adopting ADE in other contexts.

More broadly, I would like to understand in which cases lexical correction during evaluation is preferable to using a semantic evaluator. ADE cannot recover correct answers that the underlying matching system missed, and it might penalize correct explanations that happen to mention additional entities. The authors acknowledge these limitations, but the paper should explain more clearly when this trade-off is acceptable and what practical need ADE addresses.

2) The comparison with the "LLM-as-Judge" approach does not sufficiently demonstrate the value of the method relative to current alternatives.

Comparing results against human judgments is useful. However, the models used as judges—GPT-3.5-turbo, GPT-4o-mini, and DeepSeek-V3—are relatively dated, and I did not find a clear explanation as to why these specific models were selected. While the results show how ADE performs in relation to these particular models, it remains unclear whether it stays competitive against more recent and higher-performing semantic evaluators.

It would be particularly valuable to include a comparison with a recent, capable, open-weight model (e.g., Qwen3.5), even if only on a subset of the annotated responses. Since such a model could be run locally, the absence of API costs in ADE does not fully capture its practical advantage. Furthermore, the comparison should account for alignment with human judgments, execution time, hardware requirements, and computational cost. The authors correctly describe ADE as a complement to—rather than a replacement for—semantic evaluators. Nevertheless, its utility as a more cost-effective alternative would be more compelling if the paper demonstrated the trade-off in evaluation quality compared to a more robust evaluator, as well as the actual computational savings achieved.

Questions for authors
  • Which parts of ADE require a KG, and which can be constructed from a standard QA benchmark? How much performance is lost without neighborhood information? This would help clarify whether the method is mainly a KGQA-specific tool or a more general evaluation approach tested in KGQA.
  • Why were GPT-3.5-turbo, GPT-4o-mini, and DeepSeek-V3 selected as the judge baselines? Could you include a recent, capable open-weight judge (e.g., Qwen3.5) on at least a subset of the human-annotated responses? I would like to understand the trade-off in human agreement, runtime, and computational cost, rather than API cost alone.
  • How many incorrect responses does ADE correctly reject, and how many correct responses does it reject after the correction? Please report the confusion matrix before and after ADE. This is particularly important for correct responses that discuss several alternative entities before giving a clear final answer.
  • For the LLM judges, responses are truncated to 3,000 characters and at most 20 gold answers or aliases are included. Do ADE and the human annotators receive the same information? How many examples are affected by these restrictions? Since the paper focuses on verbose responses, I would like to understand whether truncation influences the comparison.
  • How stable are the results across different samples and numbers of negative controls? Fixing the controls across systems makes the comparison consistent, but does not show whether another control sample would produce similar scores and rankings.
Reviewer_TatROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose the Analytic Debiasing Evaluator (ADE), a simple, interpretable, and judge-free evaluation framework for addressing evaluator bias in KGQA tasks. The authors point out that traditional string-based evaluators are prone to two types of error when dealing with free-form generative responses: correct answers may be missed because their surface forms differ, and incorrect answers may be wrongly accepted because they mention the gold answer or one of its aliases. ADE therefore first uses alias matching to obtain high-recall candidates, and then estimates the risk of lexical collision through negative-control aliases matched to the gold answer, thereby reducing the reliability score of accepted matches accordingly.Although experiments on benchmarks such as WebQSP and CWQ verify the consistency between ADE and human judgements, as well as the soundness of its design, the paper still has considerable room for improvement in its contribution claims, the completeness of its experimental evidence, and the adequacy of its analysis. It is not yet sufficient to support the conclusion that ADE can serve as a general-purpose generative KGQA evaluator or as an alternative to LLM-as-Judge.

Critical strengths
  1. The motivation and problem are clear and reasonable. Evaluation based on exact string matching does indeed exhibit a certain degree of bias when applied to free-form generative responses, and is susceptible to issues such as the golden answer.

  2. The analysis is clearly interpretable. The paper formalises this evaluator bias and decomposes it into false-negative mismatch and false-positive lexical contamination. It also identifies two corollaries, “Ranking inversion boundary” and “Unidentifiability”, and its reasoning is rigorous.

3.Compared with LLM Judge, this method has zero API cost and is lightweight.

Critical weaknesses
  1. Although the paper decomposes string-based evaluator errors into FN and FP, it only corrects FP and does not address FN. ADE therefore essentially performs further reliability assessment only on positive cases already accepted by the base matcher, and cannot recover FNs. Although the paper acknowledges this limitation, it reduces the contribution and completeness of ADE as a new evaluation strategy. Consequently, calling it an “Evaluator” is not appropriate.

  2. In the experimental section, Table 1 primarily compares ADE with the best-performing string-based evaluator and one LLM-as-Judge baseline. ADE’s gains over the strongest existing evaluator are modest, and it does not demonstrate a consistent advantage across settings. The current evidence therefore does not sufficiently support the broader claim that ADE is a more reliable evaluator for generative KGQA or that it can serve as an alternative or complement to LLM-as-Judge. The authors could include more LLM-as-Judge and other evaluator baselines, and report results for different base-evaluator-plus-ADE combinations to establish whether ADE complements existing methods.

  3. ADE’s central claims rely on the strong assumption that gold-answer aliases and negative-control aliases are exchangeable. However, the current calibration experiments focus mainly on human-annotated samples from WebQSP. This evidence is insufficient to establish that the assumption holds consistently on CWQ, across outputs from different models and answer types, or in other tasks.

  4. The paper discusses evaluator performance primarily through aggregate metrics such as ΔEval and accuracy. However, reductions in false positives may be offset by increases in false negatives. Given that ADE does not consistently outperform existing evaluators, a more fine-grained analysis is needed, including case-level examples and stratification by question and response types. This would clarify the scenarios in which ADE is more or less effective.

Questions for authors
  1. Is ADE's false-positive correction consistent across different base evaluators? The paper currently mainly demonstrates ADE's effect on a particular base string-matching configuration, and it remains unclear whether it can stably reduce FP across different matching rules, normalisation schemes, and matching strictness. The authors are advised to further report complete results for different base evaluator + ADE combinations, and to conduct paired comparisons against the corresponding uncorrected base evaluators, analysing FP, FN, overall disagreement rate, and human agreement separately. This would test whether ADE's gains are generalisable.

  2. The magnitude of ADE's advantage is limited. In the current experiments, ADE's improvement over the best string-based evaluator is small, and in some settings it is not superior to LLM-as-Judge; the existing evidence is therefore insufficient to support a claim that it has a stable and general evaluation advantage. The authors are advised to further provide representative success and failure cases, together with stratified analyses by question type, answer type, response length, number of aliases, number of candidate answer mentions, and other factors, in order to specify concretely in which scenarios ADE can effectively reduce FP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.