acceptodds
Under review as a conference paper at ICLR 2027

Do Not Blame Your RAG System: The Multi-Hop QA Benchmarks Need Revision

Abstract

Retrieval-augmented generation (RAG) in modern agents underpins knowledge, skill, memory and tool retrieval. RAG systems are primarily evaluated on three multi-hop question answering (QA) benchmarks, namely HotpotQA, 2WikiMultiHopQA and MuSiQue, which test whether a system can retrieve the relevant documents and generate the correct answer by reasoning over them. However, we show that these benchmarks have significant flaws. On average, 39.8% of the samples carry at least one defect; for example, the question cannot be answered as written, the gold documents do not support the official answer, or the official answer is wrong. Our preliminary studies further show that even when given the gold documents, models of different sizes and families still miss the official answer on 15.9% to 24.8% of all questions, mainly on the defective samples. These defects inherently bias the evaluation of RAG systems, which rests on a flawed standard that requires urgent revision. We target this goal and release two audited versions of each benchmark: a repaired version (22,255 samples in total) based on our LLM auditing framework and a human-verified subset (2,732 samples in total). Notably, by auditing all 22,398 samples in the development sets of the three benchmarks, our LLM Auditor flags 35.6% of HotpotQA, 26.8% of 2WikiMultiHopQA and 57.1% of MuSiQue as defective under a taxonomy of twelve defect labels we define. We further evaluate the state-of-the-art RAG methods on our audited benchmarks to set up a reliable baseline for the community. We hope these released benchmarks and our framework will enable more reliable evaluation and development of future RAG systems.

Primary area:
datasets and benchmarks
Keywords:
retrieval-augmented generation, multi-hop question answering, benchmark auditing and repair

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YkCDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies defects in three widely used multi-hop QA benchmarks: HotpotQA, 2WikiMultiHopQA, and MuSiQue. The authors introduce a taxonomy of twelve defect labels and develop a semi-automatic audit and repair framework combining an LLM Auditor with human verification. By auditing 22,398 development samples, the framework identifies defects in 35.6%, 26.8%, and 57.1% of the three benchmarks, respectively. The authors release repaired benchmarks containing 22,255 samples and a human-verified benchmark containing 2,732 samples. Re-evaluation of nine RAG methods and several Oracle settings shows consistent performance improvements after repair, with most gains attributed to rewritten questions.

Critical strengths
  1. The paper addresses an important problem in multi-hop RAG evaluation and provides a systematic audit of three widely adopted benchmarks. The twelve-label taxonomy offers a useful framework for identifying benchmark defects, supported by concrete examples. The released audit records, human annotations, auditor code, and model predictions provide valuable resources for reproducibility and further research.

  2. The experimental evaluation is comprehensive, covering nine RAG methods, multiple model backbones, and Oracle settings. The authors also provide statistical confidence intervals, matched-sample comparisons, and analyses of performance gains and question difficulty. These analyses help demonstrate how benchmark defects may affect RAG evaluation and comparisons between methods.

Critical weaknesses
  1. The human verification protocol may introduce anchoring bias, as annotators read the LLM Auditor's conclusions before making their judgments. Although the reported inter-annotator agreement is high (κ = 0.884), both annotators rely on the same LLM audit, raising concerns about the independence of verification. The extent to which the Critic independently evaluates the Analyzer's conclusions also remains unclear.

  2. Most performance gains come from question rewrites. Since some repairs rewrite questions to match the available evidence, they may alter the original task or retrieval difficulty. Although Appendix G.4 examines question difficulty, retrieval-level comparisons are missing, making it difficult to fully attribute the improvements to defect correction.

Questions for authors
  1. Could a small subset of samples be independently double-annotated without access to the LLM Auditor's output, to better assess the reliability of the verification protocol?

  2. For question rewrites, could authors clarify how semantic consistency with the original questions is assessed, particularly when the queried relation changes? How do you distinguish repairing an existing question from constructing a new one?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YkCD – W1 & Q1 (Part 1/3)

Dear Reviewer YkCD,

Thank you very much for your insightful review and constructive feedback. We appreciate your recognition of our contribution to addressing important defects in three widely adopted multi-hop RAG evaluation benchmarks. We have carefully prepared a response to each of your comments below and hope it addresses your concerns.

W1 (first part) & Q1. Could a small subset of samples be double-annotated independently without access to the output of the LLM Auditor?

R1 to W1 (first part) & Q1. We appreciate this insightful comment. In response, we asked an additional annotator to independently annotate the 300 samples reported in Sec. 5.2 and Appendix D.4 without access to the LLM Auditor's outputs. Neither annotator participated in the human verification conducted for the paper. We report the agreement in terms of two metrics, raw agreement and Cohen's κ, among the original independent annotator (blind annotation 1), the new annotator (blind annotation 2) and the LLM Auditor in the table below.

ComparisonSamplesRaw agreementCohen's κ
Blind annotation 1 vs. blind annotation 23000.8300.661
LLM Auditor vs. blind annotation 13000.8300.643
LLM Auditor vs. blind annotation 23000.8200.642
LLM Auditor vs. the verdict shared by both blind annotations2490.8920.778

As shown in the above table, the agreement between the LLM Auditor and both independent human annotators is substantial (>0.61) in κ, and high regarding raw agreement (≥0.82). This further validates the reliability of our verification protocol, in addition to the single-annotator alignment reported in the submission.

Further, we updated the results in Sec. 5.2 (lines 342–345 and Table 2), Appendix D.4 (lines 1727–1733 and Table 9) and the related content to further support the reliability of our verification protocol.

W1 (second part). The extent to which the Critic independently evaluates the Analyzer's conclusions also remains unclear.

R2 to W1 (second part). We thank the reviewer for this insightful comment. In our implementation, the Critic is a separate agent with its own prompt and output format, and it is invoked separately from the Analyzer and does not share the Analyzer's conversation history (code/llm_auditor/client.py, lines 112–118 and 240–260).

In addition, our Critic is explicitly prompted to "independently decide whether the original question can be answered from the official gold evidence before judging the audit", as documented in our open-source implementation at code/llm_auditor/prompts/criticizer.md (lines 5–7).

AuthorsOpenReview ↗

Response to Reviewer YkCD – W2 & Q2 (Part 2/3)

W2 & Q2. Can the authors provide retrieval-level comparisons before and after rewriting, clarify how semantic consistency with the original questions is assessed when the queried relation changes, and explain how repairing an existing question is distinguished from constructing a new one?

R3 to W2 & Q2. We appreciate this insightful comment. We organized our response from the perspectives of Retrieval-level comparisons, Semantic consistency, Queried relation, and Repairing versus constructing a new question as follows.

Retrieval-level comparisons. We conducted an experiment to compare the retrieval difficulty of the original benchmarks with that after rewriting. Because the questions of some samples cannot be deterministically linked to their gold documents, these defective samples cannot reflect the retrieval difficulty of the original benchmarks. We therefore discarded all defective samples in the verified benchmark and kept the remaining samples as clean. We then added the rewritten samples whose gold documents remain unchanged to form clean + rewritten, reflecting the retrieval difficulty after rewriting. These two sets provide a fair comparison to measure retrieval difficulty changes before and after rewriting through the retrieval-level performance delta. Specifically, with all other experimental settings fixed, we measured the evidence-retrieval Recall@5 and the four scores of LinearRAG, HippoRAG 2 and LogicRAG on both clean and clean + rewritten sets. The first table shows that the evidence-retrieval Recall@5 stays the same from clean to clean + rewritten (65.7% → 65.7% on average). This indicates that our rewriting replaced the broken relations in the questions with the correct ones stated in the gold documents, without changing the retrieval difficulty. On top of this, the second table shows that the scores of the three methods rise in almost all cases across the four metrics and the three benchmarks. This supports a major topic of our paper that the RAG systems should not be blamed for their low scores on defective benchmarks, and highlights the necessity of fixing defects for a fair comparison and future development of the RAG systems.

Evidence-retrieval Recall@5 (%), clean → clean + rewrittenHotpotQA (609 → 729 samples)2Wiki (701 → 837 samples)MuSiQue (366 → 721 samples)Average
Average of three methods72.0 → 71.965.7 → 66.259.3 → 58.865.7 → 65.7
LinearRAG (Qwen3-32B)60.6 → 60.766.3 → 66.747.5 → 48.258.1 → 58.5
HippoRAG 2 (Qwen3-32B)92.5 → 92.576.5 → 77.878.6 → 78.882.6 → 83.0
LogicRAG (Qwen3-32B)62.9 → 62.654.4 → 54.251.8 → 49.456.4 → 55.4
Score (%), clean → clean + rewrittenHotpotQA (609 → 729 samples)2Wiki (701 → 837 samples)MuSiQue (366 → 721 samples)
Average of three methods
EM55.2 → 56.548.4 → 48.931.7 → 38.3
Cover-EM68.9 → 70.163.9 → 64.842.7 → 48.6
F165.7 → 66.757.9 → 58.343.7 → 49.1
LLM judge77.0 → 77.464.0 → 65.054.8 → 57.9
LinearRAG (Qwen3-32B)
EM33.2 → 34.418.5 → 20.214.5 → 20.5
Cover-EM59.6 → 61.054.5 → 56.030.1 → 35.9
F146.0 → 47.135.1 → 35.926.2 → 31.5
LLM judge65.0 → 65.650.8 → 52.741.0 → 44.0
HippoRAG 2 (Qwen3-32B)
EM73.1 → 74.855.6 → 57.143.2 → 50.8
Cover-EM81.9 → 83.460.2 → 62.652.7 → 59.5
F182.7 → 83.961.3 → 63.256.8 → 62.2
LLM judge90.1 → 90.763.1 → 65.465.0 → 68.5
LogicRAG (Qwen3-32B)
EM59.4 → 60.471.0 → 69.337.4 → 43.7
Cover-EM65.0 → 65.776.9 → 75.745.4 → 50.5
F168.5 → 69.177.3 → 75.948.1 → 53.6
LLM judge75.9 → 75.978.2 → 77.158.5 → 61.2

(Continued in Part 3/3)

AuthorsOpenReview ↗

Response to Reviewer YkCD – W2 & Q2 (Part 3/3)

(Continued from Part 2/3)

Semantic consistency. We conducted an experiment to verify that the rewritten questions are semantically consistent with their original ones. Following Zhu et al. [1], we used an LLM to judge whether each rewritten question is semantically consistent with its original one. For each of the 715 rewritten questions, we gave GPT-6.1-sol the original and the rewritten questions with their answers and gold documents. To avoid bias toward either question, we showed the two questions in a random order and did not tell GPT-6.1-sol which one was the original. Of the 715 rewritten questions, 92.9% are judged consistent. For all 1,056 repaired samples, GPT-6.1-sol also gave the answer type of the original and the repaired questions and judged whether their reasoning starts from the same entity. Of these samples, 95.3% keep the same answer type and 90.4% keep the same starting entity. On average, the repaired questions keep 79.7% of the words of their original ones. As reported in Appendix G.4 of the paper, the human annotators also record that 92.8% of the repaired samples keep the same number of reasoning hops. Taken together, these results show that the repaired questions stay semantically consistent with their original questions.

HotpotQA2WikiMuSiQueTotal
715 question-rewritten samples (136 / 144 / 435)
Semantically consistent93.4%99.3%90.6%92.9%
1,056 repaired samples (311 / 223 / 522)
Same answer type97.7%89.2%96.4%95.3%
Same starting entity of the reasoning95.5%96.0%85.1%90.4%
Words of the original question kept89.3%81.6%73.1%79.7%

Queried relation. We appreciate this insightful comment. We would like to clarify that our work aims to repair the reasoning relation between the question and its gold documents. We therefore prioritize an unambiguous reasoning link to the gold documents over semantic consistency between the original and rewritten questions. Meanwhile, we regularize the repaired question to be close to the original, which can be found in our implementation at code/llm_auditor/prompts/shared_rules.md (rules 1, 3 and 6, lines 66–87):

  • "Write a complete replacement question that remains as close as practical to the original question's topic, entities, requested relation, style, and difficulty."
  • "Preserve the original inference burden whenever the supplied evidence permits. Do not turn a multi-step inference question into a direct fact lookup merely to make it answerable. Do not add intermediate answers, explanatory appositives, parenthetical hints, copied evidence clauses, or extra entity descriptions that reveal the reasoning path or make retrieval unnecessary."
  • "Prefer a minimal edit that repairs the broken relation or missing linkage. When the original question is already evidence-grounded and the defect is only in its answer, aliases, or annotations, it may be retained verbatim, but still populate both question and answer as the complete repaired pair."

Repairing versus constructing a new question. In this work, we distinguish repairing an existing question from constructing a new one by considering whether the reasoning relation between the question and its gold documents comes from the original benchmark. For example:

  • If a question lacks a qualifier and thus points to more than one entity or answer (underspecified_question and ambiguous_question), we repair it by adding a qualifier that the gold documents state.
  • If a question were built from scratch with a reasoning relation that does not come from the original benchmark, we would categorize it as constructing a new question. Notably, there are no such newly constructed questions in our work because building a question from scratch would introduce semantic and difficulty-level bias, making it less manageable and less reliable.

We also added the retrieval-level comparisons and the assessment of semantic consistency to Appendix G.4 (lines 2247–2275 and Tables 24–26) in the revision. We sincerely thank the reviewer again for the valuable comments, and we hope our responses address the concerns.

Yours sincerely,

The authors of Submission 27526

[1] Zhu, Kaijie, et al. "Dynamic Evaluation of Large Language Models by Meta Probing Agents." ICML'24.

Reviewer_VHyDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies and repairs defects in three widely used multi-hop QA benchmarks: HotpotQA, 2WikiMultiHopQA, and MuSiQue. The authors propose a taxonomy of twelve defect types and develop an LLM-based auditing framework to examine all 22,398 development-set samples. They release a repaired benchmark containing 22,255 samples, along with a human-verified subset of 2,732 samples. By re-evaluating nine RAG methods, the authors show that benchmark defects can substantially underestimate performance and distort comparisons between methods. The work aims to provide a more reliable evaluation foundation for multi-hop RAG systems.

Critical strengths
  1. The benchmark and resource contribution is substantial. The audit covers more than 22,000 samples across three widely used benchmarks, and the defect taxonomy, repaired datasets, and human-verified subset should be useful to others working on RAG evaluation.

  2. The empirical work is broad. Automated auditing, human verification, and re-evaluation of nine RAG methods are combined in one study, and the results show that benchmark defects affect reported numbers and can change the apparent gap between methods. The extra analyses on question rewriting and reasoning depth add useful support.

Critical weaknesses
  1. Question rewriting may change the task being evaluated. A large share of questions are rewritten, especially in MuSiQue (48%), and rewritten samples account for most of the reported gains (88% on average across the repaired benchmarks). The authors do check reasoning depth, answer leakage, and closed-book accuracy, but keeping the same number of hops does not guarantee that retrieval difficulty or the original reasoning demands are preserved. So it is still unclear how much of the improvement comes from fixing benchmark defects and how much comes from changing the questions themselves.
Questions for authors
  1. Preserving task difficulty. Beyond hop count and closed-book accuracy, what evidence can the authors give that rewriting preserves retrieval and reasoning difficulty? For instance, how does evidence-retrieval recall change before and after rewriting, and do the required reasoning relations stay the same?

  2. Interpreting the performance gains. Since most gains come from rewritten samples, can the authors separate more clearly the improvements due to corrected labels from those due to changes in the questions or in retrieval behavior?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VHyD – W1 (first part) & Q1 (Part 1/2)

Dear Reviewer VHyD,

We appreciate your recognition that our work provides useful resources and supports efforts towards more reliable RAG evaluation. We hope the responses below address your concerns.

W1 (first part) & Q1. Can the authors provide additional evidence that question rewriting preserves retrieval and reasoning difficulty?

R1 to W1 (first part) & Q1. We appreciate the reviewer for this insightful comment. We conducted experiments to assess whether question rewriting preserves the retrieval and reasoning difficulty of the original benchmarks, as detailed below.

Evidence-retrieval recall. Most importantly, we agree that evidence-retrieval recall can indicate relative retrieval difficulty, provided that the questions are deterministically linked to their gold documents. However, the questions of some samples cannot be deterministically linked to their gold documents, so these samples cannot reflect the retrieval difficulty of the original benchmarks. We therefore discarded all defective samples in the verified benchmark and kept the remaining samples as clean. We then added the rewritten samples whose gold documents remain unchanged to form clean + rewritten, reflecting the retrieval difficulty after rewriting. The results show that the average evidence-retrieval Recall@5 was unchanged between the two sets (65.7% in both). This result suggests that the rewritten samples have similar retrieval difficulty to the clean ones under this metric.

Evidence-retrieval Recall@5 (%), clean → clean + rewrittenHotpotQA (609 → 729 samples)2Wiki (701 → 837 samples)MuSiQue (366 → 721 samples)Average
Average of three methods72.0 → 71.965.7 → 66.259.3 → 58.865.7 → 65.7
LinearRAG (Qwen3-32B)60.6 → 60.766.3 → 66.747.5 → 48.258.1 → 58.5
HippoRAG 2 (Qwen3-32B)92.5 → 92.576.5 → 77.878.6 → 78.882.6 → 83.0
LogicRAG (Qwen3-32B)62.9 → 62.654.4 → 54.251.8 → 49.456.4 → 55.4

Meanwhile, regarding your question about the recall changes before and after rewriting, we report Recall@5 of the same three methods on the original and the rewritten questions in the table below. Recall@5 rises as expected after rewriting (60.4% → 66.3% on average), because our rewriting replaces the incorrect relations in the original questions with the correct ones.

Evidence-retrieval Recall@5 (%), before → after rewritingHotpotQA2WikiMuSiQueAverage
Gold documents unchanged (120 / 136 / 355 samples)
Average of three methods67.6 → 71.764.0 → 68.949.5 → 58.360.4 → 66.3
LinearRAG (Qwen3-32B)58.3 → 61.265.1 → 68.842.0 → 48.955.1 → 59.6
HippoRAG 2 (Qwen3-32B)87.5 → 92.576.3 → 84.668.3 → 78.977.4 → 85.3
LogicRAG (Qwen3-32B)57.1 → 61.250.6 → 53.338.3 → 47.048.6 → 53.8

Required reasoning relations. Our rewriting replaces the incorrect reasoning relations with those supported by the gold documents. We rewrite a question only when it cannot be unambiguously linked to its gold documents. We also enforce strict rules for both the LLM Auditor and human annotators when proposing rewritten questions, as documented in our open-source implementation at code/llm_auditor/prompts/shared_rules.md (rules 1, 3 and 6, lines 66–87):

  • "Write a complete replacement question that remains as close as practical to the original question's topic, entities, requested relation, style, and difficulty."
  • "Preserve the original inference burden whenever the supplied evidence permits. Do not turn a multi-step inference question into a direct fact lookup merely to make it answerable. Do not add intermediate answers, explanatory appositives, parenthetical hints, copied evidence clauses, or extra entity descriptions that reveal the reasoning path or make retrieval unnecessary."
  • "Prefer a minimal edit that repairs the broken relation or missing linkage. When the original question is already evidence-grounded and the defect is only in its answer, aliases, or annotations, it may be retained verbatim, but still populate both question and answer as the complete repaired pair."
AuthorsOpenReview ↗

Response to Reviewer VHyD – W1 (second part) & Q2 (Part 2/2)

W1 (second part) & Q2. Can the authors separate the improvements due to corrected evaluation labels from those due to changes in the questions or in retrieval behavior?

R2 to W1 (second part) & Q2. We appreciate this insightful comment. In order to fully address it, we conducted an additional experiment to attribute the overall performance gains to each of the actual repairs of the question (Q), the answer or its aliases (A), the gold document marks (D) and their combinations. This experiment uses the 1,056 repaired samples of the verified benchmark. Many samples carry more than one defect, so their repair often changes more than one field. The results are presented in the table below, in which each cell gives the performance gain from each repair, averaged over the 12 RAG rows of Table 3, with the number of samples in parentheses. By comparing columns A, Q and D, we see that there are more cases with revised answers than rewritten questions or corrected gold documents, leading to more performance gains on HotpotQA and MuSiQue, but the RAG methods benefit more from rewritten questions on 2Wiki. Such a difference is expected as these datasets exhibit different defects (Sec. 5.1 and 5.2). For the combined repairs in other columns, revising both the question and answer contributes most to the improvement, especially on MuSiQue. Although the contribution varies between our proposed repairs, their non-negative impact validates our efforts.

DatasetAQDQ+AQ+DA+DQ+A+D
HotpotQA+0.43 (167)0.00 (0)0.00 (0)+2.71 (120)0.00 (0)+0.25 (8)+0.52 (16)
2Wiki+0.74 (78)+1.97 (60)0.00 (1)+2.15 (76)+0.05 (2)0.00 (0)+0.26 (6)
MuSiQue+0.10 (80)0.00 (0)0.00 (0)+8.07 (355)0.00 (0)+0.02 (7)+3.26 (80)

We also added the comparison of retrieval difficulty in R1 to Appendix G.4 (lines 2259–2275 and Table 25) and the attribution of the gains in R2 to Appendix G.3 (lines 2152–2161 and Table 20) in the revision. We hope our answers fully address your concerns, and we are happy to answer any follow-up questions.

Yours sincerely,

The authors of Submission 27526

Dear Reviewer VHyD,

Thank you again for your detailed review comments. We hope our detailed response fully addresses your concerns, and we appreciate the time and effort you put into reviewing our paper.

Yours sincerely,

The authors of Submission 27526

Reviewer_arKcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits the dev sets of HotpotQA, 2WikiMultiHopQA and MuSiQue, the three benchmarks most multi-hop RAG work reports on. It defines twelve defect categories and uses GPT-5.6 as an auditor to label and repair incorrect data points. Two annotators then review 1,000 random samples per benchmark. The auditor flags 35.6%, 26.8% and 57.1% of samples as defective, most often because no supplied document states a needed fact. The authors release a repaired version (22,255 samples, auditor only) and a human-verified subset (2,732 samples). Nine RAG methods score higher than reported on the released versions.

Critical strengths
  1. A useful resource for benchmarks the field relies on. The full-dev-set repair of these benchmarks is very meaningful, especially audited by human annotators, two-annotator subset, per-sample audit and review records, and code.
  2. The re-evaluation is thorough and shows that the repairs recover the lost points, showing the effectiveness of the repairing approach.
Critical weaknesses
  1. The leading label, required_evidence_absent, is judged against each task's provided 10-20 documents. The RAG evaluation, however, retrieves from a pool of all the dataset's paragraphs, it is possible that the supported evidence is incorrectly annotated, but the task itself is still valid within the corpus.
  2. It is unclear how similar the repaired tasks are to the original ones. The paper checks hop counts, answer leakage and closed-book accuracy. Yet, it will be more helpful to show that the fixed tasks still satisfies the desiderata of from the original datasets, i.e. answer type and retrieval difficulty.
Questions for authors
  1. For required_evidence_absent samples, how often does another paragraph in the pooled corpus state the missing fact? How do the gains change if those samples keep their original question?
  2. Can you show a random set of original vs rewritten questions, and report some metric around how big is the fix, and whether it keeps the original dataset's design intention?
4 comments
AuthorsOpenReview ↗

Response to Reviewer arKc – W1 & Q1 (Part 1/2)

Dear Reviewer arKc,

Thank you very much for your thorough review and constructive feedback. We appreciate that you find our benchmark repair and re-evaluation meaningful. Per your comments, we carefully prepared the following responses and hope they address your concerns.

W1 & Q1 (first part). For required_evidence_absent samples, how often does another document in the pooled corpus state the missing fact?

R1 to W1 & Q1 (first part). We thank the reviewer for this insightful question. We agree that other documents in the pool might state the missing fact, and we conducted an additional experiment to validate such cases. As shown in the table below, we found that required_evidence_absent samples whose missing fact is stated in another document of the pool account for less than 1%.

Specifically, we checked all 514 samples labeled as required_evidence_absent in our verified benchmark. First, we carefully prompted GPT-6.1-sol to read the original question, the supplied documents for each sample and the original answer, and then check whether any other documents in the pool state the missing fact. Finally, we manually verified all the cases that GPT-6.1-sol identified. It's worth noting that a case is deemed as found only if a document in the pool indeed states the missing fact and makes the original question answerable.

SamplesHotpotQA2WikiMuSiQueTotal
required_evidence_absent samples64113337514
Missing fact found in the pool0134 (0.8%)
Not found64112334510 (99.2%)

For the 510 samples (99.2%) in the Not found row, no document in the pool states the missing fact, so their original questions remain unanswerable.

For the remaining 4 samples (0.8%), whose missing fact is found in the pool, we can either keep the original question and use the supporting document in the pool as its gold document, or adopt our question-rewriting strategy. We believe both are valid, and we keep the rewritten questions since the overall difficulty of the benchmarks remains at the same level, as shown in our response R3 to W2 & Q2 below.

Q1 (second part). How do the gains change if these samples keep their original question?

R2 to Q1 (second part). We thank the reviewer for this constructive suggestion. We conducted an additional experiment on the 4 aforementioned samples in R1 whose missing fact is found in the pool, and the results below show that the gains stay almost the same when we keep the original questions in these samples.

Also, for these samples, we restored the original questions and answers and left all other samples unchanged in the verified benchmark. We then recomputed each method's accuracy gain from the original benchmarks to the verified benchmark, following Table 3 of the paper. The gains changed by at most 0.2 percentage points on each benchmark, and the gains of all 12 RAG methods remain positive on all three benchmarks. Therefore, the gains in Table 3 are largely unchanged when samples whose missing facts are found in the pool retain their original questions and answers.

Gain (Table 3 → original question kept)Base LLMHotpotQA2WikiMuSiQue
Retrieval structure
LinearRAGQwen3-32B+4.2 → +4.2+4.7 → +4.6+11.9 → +12.0
GraphRAGQwen3-32B+5.1 → +5.1+6.8 → +6.8+12.9 → +12.9
HippoRAG 2Qwen3-32B+5.6 → +5.6+7.8 → +7.8+15.0 → +14.9
HippoRAG 2GPT-5.6+5.4 → +5.4+4.2 → +4.2+11.9 → +11.9
Workflow
LogicRAGQwen3-32B+4.2 → +4.2+6.6 → +6.4+11.7 → +11.9
STRIDEQwen3-32B+4.7 → +4.7+5.4 → +5.4+10.9 → +10.9
IRCoTQwen3-32B+4.2 → +4.2+7.0 → +7.0+13.0 → +13.0
IRCoTGPT-5.6+5.0 → +5.0+4.1 → +4.1+11.8 → +11.6
Agentic search
Search-R1SearchR1-7B+3.5 → +3.5+2.3 → +2.2+11.3 → +11.3
ReSearchReSearch-32B+6.3 → +6.3+6.5 → +6.5+14.7 → +14.8
Search-o1Qwen3-32B+5.3 → +5.3+6.7 → +6.7+13.5 → +13.5
Search-o1GPT-5.6+3.9 → +3.9+3.5 → +3.5+12.9 → +13.0
AuthorsOpenReview ↗

Response to Reviewer arKc – W2 & Q2 (Part 2/2)

W2 & Q2. How similar are the repaired tasks to the original ones? Do the fixed tasks maintain the answer type, retrieval difficulty and design intention of the original datasets? Can the authors report metrics around how big the fix is?

R3 to W2 & Q2. We thank the reviewer for this valuable comment. When building the verified benchmark, we devoted our best effort to preserving the answer types and difficulty levels so as to minimize the impact of our repairs on the design intention. Based on your comment, we conducted additional analysis on the repaired samples.

The results are as follows:

  • Repaired questions have the same answer type (95.3%) and keep most words of the original questions (79.7%). Specifically, for each repaired sample, we presented GPT-6.1-sol with the original and the repaired questions in random order, without indicating which was which. Each question was accompanied by its answer and gold documents, and the model was asked to identify its answer type. We also measured the proportion of the words of each original question retained in its repaired question, and the average was 79.7%.

  • Retrieval difficulty stays close to that of the original benchmarks. We conducted an experiment to assess whether question rewriting changes retrieval difficulty. Since the questions in some samples cannot be unambiguously linked to their gold documents, these defective samples cannot reflect the retrieval difficulty of the original benchmarks. We therefore discarded all defective samples in the verified benchmark and kept the remaining samples as clean. We then added the rewritten samples whose gold documents remain unchanged to form clean + rewritten, reflecting the retrieval difficulty after rewriting. The results show that the average evidence-retrieval Recall@5 was unchanged between the two sets (65.7% in both). This result suggests that the rewritten samples have similar retrieval difficulty to the clean ones under this metric.

Evidence-retrieval Recall@5 (%), clean → clean + rewrittenHotpotQA (609 → 729 samples)2Wiki (701 → 837 samples)MuSiQue (366 → 721 samples)Average
Average of three methods72.0 → 71.965.7 → 66.259.3 → 58.865.7 → 65.7
LinearRAG (Qwen3-32B)60.6 → 60.766.3 → 66.747.5 → 48.258.1 → 58.5
HippoRAG 2 (Qwen3-32B)92.5 → 92.576.5 → 77.878.6 → 78.882.6 → 83.0
LogicRAG (Qwen3-32B)62.9 → 62.654.4 → 54.251.8 → 49.456.4 → 55.4
  • The design intention of the original benchmarks is kept (92.9% semantically consistent). We prompted GPT-6.1-sol to assess whether each rewritten question was semantically consistent with its original question, and it judged 92.9% of the 715 rewritten samples to be consistent. The model also judged whether the reasoning of the original and the repaired questions starts from the same entity, and 90.4% of the repaired samples keep the same starting entity. Human annotators also recorded that 92.8% of the repaired samples keep the same number of reasoning hops, as reported in Appendix G.4. Together with the preserved answer types and the similar retrieval difficulty reported above, the repaired samples still preserve the desiderata of the original benchmarks.
HotpotQA2WikiMuSiQueTotal
Repaired samples of the verified benchmark3112235221,056
Words of the original question kept89.3%81.6%73.1%79.7%
Same answer type97.7%89.2%96.4%95.3%
Same starting entity of the reasoning95.5%96.0%85.1%90.4%
Semantically consistent (question-rewritten samples only)93.4%99.3%90.6%92.9%

We have also added the comparison of the repaired and the original questions to Appendix G.4 (lines 2247–2275 and Tables 24–25) and the experiments in R1 and R2 to Appendix G.5 (lines 2397–2422 and Tables 27–28) of the revised paper, marked in blue, and hope our answers fully address your concerns.

We thank Reviewer arKc again for the constructive and insightful review, and we are happy to answer any further questions.

Yours sincerely,

The authors of Submission 27526

Thanks for the response by the authors, I am not able to increase my score to 4 and I will maintain my score of 3

Dear Reviewer arKc,

Thanks very much for your response and for maintaining the positive rating. We value your supportive and constructive comments.

Yours sincerely,

The authors of Submission 27526

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.