- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies defects in three widely used multi-hop QA benchmarks: HotpotQA, 2WikiMultiHopQA, and MuSiQue. The authors introduce a taxonomy of twelve defect labels and develop a semi-automatic audit and repair framework combining an LLM Auditor with human verification. By auditing 22,398 development samples, the framework identifies defects in 35.6%, 26.8%, and 57.1% of the three benchmarks, respectively. The authors release repaired benchmarks containing 22,255 samples and a human-verified benchmark containing 2,732 samples. Re-evaluation of nine RAG methods and several Oracle settings shows consistent performance improvements after repair, with most gains attributed to rewritten questions.
Critical strengths
-
The paper addresses an important problem in multi-hop RAG evaluation and provides a systematic audit of three widely adopted benchmarks. The twelve-label taxonomy offers a useful framework for identifying benchmark defects, supported by concrete examples. The released audit records, human annotations, auditor code, and model predictions provide valuable resources for reproducibility and further research.
-
The experimental evaluation is comprehensive, covering nine RAG methods, multiple model backbones, and Oracle settings. The authors also provide statistical confidence intervals, matched-sample comparisons, and analyses of performance gains and question difficulty. These analyses help demonstrate how benchmark defects may affect RAG evaluation and comparisons between methods.
Critical weaknesses
-
The human verification protocol may introduce anchoring bias, as annotators read the LLM Auditor's conclusions before making their judgments. Although the reported inter-annotator agreement is high (κ = 0.884), both annotators rely on the same LLM audit, raising concerns about the independence of verification. The extent to which the Critic independently evaluates the Analyzer's conclusions also remains unclear.
-
Most performance gains come from question rewrites. Since some repairs rewrite questions to match the available evidence, they may alter the original task or retrieval difficulty. Although Appendix G.4 examines question difficulty, retrieval-level comparisons are missing, making it difficult to fully attribute the improvements to defect correction.
Questions for authors
-
Could a small subset of samples be independently double-annotated without access to the LLM Auditor's output, to better assess the reliability of the verification protocol?
-
For question rewrites, could authors clarify how semantic consistency with the original questions is assessed, particularly when the queried relation changes? How do you distinguish repairing an existing question from constructing a new one?
3 comments
Response to Reviewer YkCD – W2 & Q2 (Part 2/3)
W2 & Q2. Can the authors provide retrieval-level comparisons before and after rewriting, clarify how semantic consistency with the original questions is assessed when the queried relation changes, and explain how repairing an existing question is distinguished from constructing a new one?
R3 to W2 & Q2. We appreciate this insightful comment. We organized our response from the perspectives of Retrieval-level comparisons, Semantic consistency, Queried relation, and Repairing versus constructing a new question as follows.
Retrieval-level comparisons. We conducted an experiment to compare the retrieval difficulty of the original benchmarks with that after rewriting. Because the questions of some samples cannot be deterministically linked to their gold documents, these defective samples cannot reflect the retrieval difficulty of the original benchmarks. We therefore discarded all defective samples in the verified benchmark and kept the remaining samples as clean. We then added the rewritten samples whose gold documents remain unchanged to form clean + rewritten, reflecting the retrieval difficulty after rewriting. These two sets provide a fair comparison to measure retrieval difficulty changes before and after rewriting through the retrieval-level performance delta. Specifically, with all other experimental settings fixed, we measured the evidence-retrieval Recall@5 and the four scores of LinearRAG, HippoRAG 2 and LogicRAG on both clean and clean + rewritten sets. The first table shows that the evidence-retrieval Recall@5 stays the same from clean to clean + rewritten (65.7% → 65.7% on average). This indicates that our rewriting replaced the broken relations in the questions with the correct ones stated in the gold documents, without changing the retrieval difficulty. On top of this, the second table shows that the scores of the three methods rise in almost all cases across the four metrics and the three benchmarks. This supports a major topic of our paper that the RAG systems should not be blamed for their low scores on defective benchmarks, and highlights the necessity of fixing defects for a fair comparison and future development of the RAG systems.
| Evidence-retrieval Recall@5 (%), clean → clean + rewritten | HotpotQA (609 → 729 samples) | 2Wiki (701 → 837 samples) | MuSiQue (366 → 721 samples) | Average |
|---|---|---|---|---|
| Average of three methods | 72.0 → 71.9 | 65.7 → 66.2 | 59.3 → 58.8 | 65.7 → 65.7 |
| LinearRAG (Qwen3-32B) | 60.6 → 60.7 | 66.3 → 66.7 | 47.5 → 48.2 | 58.1 → 58.5 |
| HippoRAG 2 (Qwen3-32B) | 92.5 → 92.5 | 76.5 → 77.8 | 78.6 → 78.8 | 82.6 → 83.0 |
| LogicRAG (Qwen3-32B) | 62.9 → 62.6 | 54.4 → 54.2 | 51.8 → 49.4 | 56.4 → 55.4 |
| Score (%), clean → clean + rewritten | HotpotQA (609 → 729 samples) | 2Wiki (701 → 837 samples) | MuSiQue (366 → 721 samples) |
|---|---|---|---|
| Average of three methods | |||
| EM | 55.2 → 56.5 | 48.4 → 48.9 | 31.7 → 38.3 |
| Cover-EM | 68.9 → 70.1 | 63.9 → 64.8 | 42.7 → 48.6 |
| F1 | 65.7 → 66.7 | 57.9 → 58.3 | 43.7 → 49.1 |
| LLM judge | 77.0 → 77.4 | 64.0 → 65.0 | 54.8 → 57.9 |
| LinearRAG (Qwen3-32B) | |||
| EM | 33.2 → 34.4 | 18.5 → 20.2 | 14.5 → 20.5 |
| Cover-EM | 59.6 → 61.0 | 54.5 → 56.0 | 30.1 → 35.9 |
| F1 | 46.0 → 47.1 | 35.1 → 35.9 | 26.2 → 31.5 |
| LLM judge | 65.0 → 65.6 | 50.8 → 52.7 | 41.0 → 44.0 |
| HippoRAG 2 (Qwen3-32B) | |||
| EM | 73.1 → 74.8 | 55.6 → 57.1 | 43.2 → 50.8 |
| Cover-EM | 81.9 → 83.4 | 60.2 → 62.6 | 52.7 → 59.5 |
| F1 | 82.7 → 83.9 | 61.3 → 63.2 | 56.8 → 62.2 |
| LLM judge | 90.1 → 90.7 | 63.1 → 65.4 | 65.0 → 68.5 |
| LogicRAG (Qwen3-32B) | |||
| EM | 59.4 → 60.4 | 71.0 → 69.3 | 37.4 → 43.7 |
| Cover-EM | 65.0 → 65.7 | 76.9 → 75.7 | 45.4 → 50.5 |
| F1 | 68.5 → 69.1 | 77.3 → 75.9 | 48.1 → 53.6 |
| LLM judge | 75.9 → 75.9 | 78.2 → 77.1 | 58.5 → 61.2 |
(Continued in Part 3/3)
Response to Reviewer YkCD – W2 & Q2 (Part 3/3)
(Continued from Part 2/3)
Semantic consistency. We conducted an experiment to verify that the rewritten questions are semantically consistent with their original ones. Following Zhu et al. [1], we used an LLM to judge whether each rewritten question is semantically consistent with its original one. For each of the 715 rewritten questions, we gave GPT-6.1-sol the original and the rewritten questions with their answers and gold documents. To avoid bias toward either question, we showed the two questions in a random order and did not tell GPT-6.1-sol which one was the original. Of the 715 rewritten questions, 92.9% are judged consistent. For all 1,056 repaired samples, GPT-6.1-sol also gave the answer type of the original and the repaired questions and judged whether their reasoning starts from the same entity. Of these samples, 95.3% keep the same answer type and 90.4% keep the same starting entity. On average, the repaired questions keep 79.7% of the words of their original ones. As reported in Appendix G.4 of the paper, the human annotators also record that 92.8% of the repaired samples keep the same number of reasoning hops. Taken together, these results show that the repaired questions stay semantically consistent with their original questions.
| HotpotQA | 2Wiki | MuSiQue | Total | |
|---|---|---|---|---|
| 715 question-rewritten samples (136 / 144 / 435) | ||||
| Semantically consistent | 93.4% | 99.3% | 90.6% | 92.9% |
| 1,056 repaired samples (311 / 223 / 522) | ||||
| Same answer type | 97.7% | 89.2% | 96.4% | 95.3% |
| Same starting entity of the reasoning | 95.5% | 96.0% | 85.1% | 90.4% |
| Words of the original question kept | 89.3% | 81.6% | 73.1% | 79.7% |
Queried relation. We appreciate this insightful comment. We would like to clarify that our work aims to repair the reasoning relation between the question and its gold documents. We therefore prioritize an unambiguous reasoning link to the gold documents over semantic consistency between the original and rewritten questions. Meanwhile, we regularize the repaired question to be close to the original, which can be found in our implementation at code/llm_auditor/prompts/shared_rules.md (rules 1, 3 and 6, lines 66–87):
- "Write a complete replacement
questionthat remains as close as practical to the original question's topic, entities, requested relation, style, and difficulty." - "Preserve the original inference burden whenever the supplied evidence permits. Do not turn a multi-step inference question into a direct fact lookup merely to make it answerable. Do not add intermediate answers, explanatory appositives, parenthetical hints, copied evidence clauses, or extra entity descriptions that reveal the reasoning path or make retrieval unnecessary."
- "Prefer a minimal edit that repairs the broken relation or missing linkage. When the original question is already evidence-grounded and the defect is only in its answer, aliases, or annotations, it may be retained verbatim, but still populate both
questionandansweras the complete repaired pair."
Repairing versus constructing a new question. In this work, we distinguish repairing an existing question from constructing a new one by considering whether the reasoning relation between the question and its gold documents comes from the original benchmark. For example:
- If a question lacks a qualifier and thus points to more than one entity or answer (
underspecified_questionandambiguous_question), we repair it by adding a qualifier that the gold documents state. - If a question were built from scratch with a reasoning relation that does not come from the original benchmark, we would categorize it as constructing a new question. Notably, there are no such newly constructed questions in our work because building a question from scratch would introduce semantic and difficulty-level bias, making it less manageable and less reliable.
We also added the retrieval-level comparisons and the assessment of semantic consistency to Appendix G.4 (lines 2247–2275 and Tables 24–26) in the revision. We sincerely thank the reviewer again for the valuable comments, and we hope our responses address the concerns.
Yours sincerely,
The authors of Submission 27526
[1] Zhu, Kaijie, et al. "Dynamic Evaluation of Large Language Models by Meta Probing Agents." ICML'24.
Response to Reviewer YkCD – W1 & Q1 (Part 1/3)
Dear Reviewer YkCD,
Thank you very much for your insightful review and constructive feedback. We appreciate your recognition of our contribution to addressing important defects in three widely adopted multi-hop RAG evaluation benchmarks. We have carefully prepared a response to each of your comments below and hope it addresses your concerns.
W1 (first part) & Q1. Could a small subset of samples be double-annotated independently without access to the output of the LLM Auditor?
R1 to W1 (first part) & Q1. We appreciate this insightful comment. In response, we asked an additional annotator to independently annotate the 300 samples reported in Sec. 5.2 and Appendix D.4 without access to the LLM Auditor's outputs. Neither annotator participated in the human verification conducted for the paper. We report the agreement in terms of two metrics, raw agreement and Cohen's κ, among the original independent annotator (blind annotation 1), the new annotator (blind annotation 2) and the LLM Auditor in the table below.
As shown in the above table, the agreement between the LLM Auditor and both independent human annotators is substantial (>0.61) in κ, and high regarding raw agreement (≥0.82). This further validates the reliability of our verification protocol, in addition to the single-annotator alignment reported in the submission.
Further, we updated the results in Sec. 5.2 (lines 342–345 and Table 2), Appendix D.4 (lines 1727–1733 and Table 9) and the related content to further support the reliability of our verification protocol.
W1 (second part). The extent to which the Critic independently evaluates the Analyzer's conclusions also remains unclear.
R2 to W1 (second part). We thank the reviewer for this insightful comment. In our implementation, the Critic is a separate agent with its own prompt and output format, and it is invoked separately from the Analyzer and does not share the Analyzer's conversation history (code/llm_auditor/client.py, lines 112–118 and 240–260).
In addition, our Critic is explicitly prompted to "independently decide whether the original question can be answered from the official gold evidence before judging the audit", as documented in our open-source implementation at code/llm_auditor/prompts/criticizer.md (lines 5–7).