- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the author proposed a retrieval framework that explicitly targets evidence-chain completeness for multi-hop RAG. Specifically, the author adopts a hierarchical hypergraph, aligning everything in a unified retrieval space. At inference time, the proposed method parses the query into an evidence-chain goal and derives a compact, query-specific local relation graph for subsequent path search. The proposed method achieved the best overall performance on some benchmarks.
Critical strengths
- The paper revealed and investigated an interesting observation that there is a gap between evidence recall and complete evidence-chain coverage.
- The method design is complete and intuitive for solving the problem.
- The experiments are comprehensive, covering different types of Table-QA benchmarks, and multiple analyses are provided.
Critical weaknesses
- My main concern is the importance of the complete evidence chain, since the final performance improvement is slight and ECG-RAG does not achieve the best results on every metric. I was wondering how much an incomplete evidence chain affects QA accuracy or downstream tasks. The authors show that even with high recall, complete evidence-chain coverage is low, and in the main results, ECG-RAG sometimes achieves nearly identical accuracy to baselines.
- The method consists of multiple parts and the ablation is coarse-grained and only on one dataset; the necessity of some designs is not well justified. For example, are both Ant-inspired Exploration and A-style Search necessary? Can we replace them with a simpler method?
- If the evidence-chain goal generated by an LLM is itself incomplete or incorrect, can subsequent searches still retrieve the correct evidence?
- The method consists of multiple hyperparameters, but some key hyperparameter values are missing, which affects the reproducibility and clarity of the method. For example, the values of alpha, beta, and others are not specified.
Questions for authors
Please see the weaknesses.
9 comments
Response to Reviewer Tp3R (Part 2/8)
Q1&W1. Importance of complete evidence chains for downstream QA.
Controlled evidence-coverage analysis.
2. Fine-grained analysis of evidence recall and completeness. Using the existing answer scores for the same 234 questions, we formed weighted mixtures of the zero-to-four supporting-fact retention levels to examine average ER and CCC from 0% to 75% in 15-percentage-point increments. For each target pair, the weight on the 4/4 condition equals CCC/100; the remaining weight is distributed between adjacent incomplete-retention levels to match the target average ER. The same weights are applied to every question.
EM (%).
| Average ER (%) / CCC (%) | 0 | 15 | 30 | 45 | 60 | 75 |
|---|---|---|---|---|---|---|
| 0 | 2.14 | |||||
| 15 | 3.93 | 15.85 | ||||
| 30 | 6.50 | 17.65 | 29.57 | |||
| 45 | 10.60 | 20.79 | 31.37 | 43.29 | ||
| 60 | 17.26 | 25.53 | 35.09 | 45.28 | 57.01 | |
| 75 | 25.21 | 33.48 | 41.75 | 50.02 | 59.57 | 70.73 |
F1 (%).
| Average ER (%) / CCC (%) | 0 | 15 | 30 | 45 | 60 | 75 |
|---|---|---|---|---|---|---|
| 0 | 2.42 | |||||
| 15 | 4.22 | 16.10 | ||||
| 30 | 6.78 | 17.89 | 29.77 | |||
| 45 | 10.88 | 21.03 | 31.57 | 43.45 | ||
| 60 | 17.55 | 25.78 | 35.28 | 45.43 | 57.12 | |
| 75 | 25.50 | 33.73 | 41.95 | 50.18 | 59.69 | 70.80 |
Rows indicate average ER and columns indicate CCC; blank cells represent infeasible combinations rather than zero scores.
Vertical comparisons hold CCC fixed and show that increasing ER improves both EM and F1, confirming the value of partial evidence. Horizontal comparisons hold average ER fixed and show the benefit of increasing complete-chain coverage by changing how evidence is distributed. Under this weighting scheme, adjacent 15-percentage-point increases in CCC at fixed average ER yield larger EM and F1 gains than adjacent 15-percentage-point increases in ER at fixed CCC throughout the displayed grid. These patterns show that, for these multi-hop questions and controlled contexts, evidence completeness provides important information beyond average recall, supporting CCC as a complementary diagnostic alongside ER.
3. QA performance across all evidence-retention levels. We further examined the full range from zero to four retained supporting facts on the same 234 questions:
| Supporting facts retained per question | Average ER (%) | CCC (%) | EM (%) | F1 (%) |
|---|---|---|---|---|
| 0/4 | 0.0 | 0.0 | 2.14 | 2.42 |
| 1/4 | 25.0 | 0.0 | 5.13 | 5.41 |
| 2/4 | 50.0 | 0.0 | 11.97 | 12.25 |
| 3/4 | 75.0 | 0.0 | 25.21 | 25.50 |
| 4/4 | 100.0 | 100.0 | 93.59 | 93.59 |
Evidence subsets follow the same nested retention order. The 0/4 condition contains matched distractors rather than an empty context.
This breakdown reveals two complementary patterns. First, additional partial evidence improves QA even when CCC remains zero: moving from 2/4 to 3/4 supporting facts increases both EM and F1 by 13.25 points. Second, completing the support set produces a substantially larger gain: moving from 3/4 to 4/4 increases EM by 68.38 points and F1 by 68.09 points. Both transitions add one supporting fact and increase ER by 25 percentage points.
Together, the fixed-ER comparison and the retention-level breakdown show why evidence completeness matters for these multi-hop questions: partial evidence provides useful information, while recovering the remaining supporting fact can enable an answer that the incomplete context does not support.
Response to Reviewer Tp3R (Part 3/8)
Q1&W1. Importance of complete evidence chains for downstream QA.
Controlled evidence-coverage analysis.
4. Supporting-sentence deletion in actual generation contexts. We also analyzed 850 questions whose original ECG-RAG contexts covered all annotated supporting sentences. We compared deleting a selected supporting sentence and its exact duplicates with deleting matched non-gold sentence spans. Other context content and generation instructions were preserved, with final context lengths constrained to differ by at most 32 tokens and 0.5% of the original length.
| Condition | ER (%) | CCC (%) | EM (%) | F1 (%) |
|---|---|---|---|---|
| Matched non-gold sentence deletion | 100.00 | 100.00 | 64.94 | 72.86 |
| Supporting-sentence deletion | 55.62 | 1.53 | 51.06 | 56.66 |
Paired interventions were successfully constructed for 837 questions and both conditions were regenerated. The remaining 13 questions retain identical historical predictions in both conditions and contribute zero paired effect; all 850 questions are included in the analysis.
Deleting supporting sentences reduces EM by 13.88 points and F1 by 16.20 points relative to the matched deletion control. The respective 95% confidence intervals are [11.18, 16.59] and [13.54, 18.93], with Holm-adjusted for both metrics.
Both analyses use 20,000 question-paired bootstrap resamples, two-sided tests, and Holm correction within each analysis. Together, these interventions show that missing supporting evidence can substantially reduce downstream QA accuracy, including in ECG-RAG’s actual generation contexts. These findings provide empirical support for evaluating evidence completeness alongside average recall in multi-hop QA.
System comparison under a common context budget. We also conducted an additional comparison on 1,000 2WikiMultiHopQA questions with both methods subject to a 5,000-token final-context budget. ECG-RAG uses compact packing that removes redundant instructions and retrieval diagnostics while retaining relation-path content and packing supporting text within the budget.
| Method | Context budget | Average context tokens | EM | F1 |
|---|---|---|---|---|
| IGMiRAG | 5,000 | 4,740.53 | 57.30 | 66.12 |
| ECG-RAG | 5,000 | 4,806.54 | 59.20 | 67.46 |
ECG-RAG improves EM/F1 by 1.90/1.34 percentage points, while the actual average context lengths differ by 66.01 tokens, approximately 1.4%. This comparison connects the evidence-availability analyses above to end-to-end system performance under a shared context budget. The intervention experiments quantify sensitivity to missing evidence; this experiment measures the QA difference between retrieval systems.
Why the gains differ across task types. ECG-RAG primarily targets multi-hop QA, where answering requires connecting distributed facts through intermediate entities or relations. Missing one necessary link can leave a question unsupported even when other retrieved passages are relevant. Explanatory QA instead requires broader information synthesis and coherent descriptions, for which a single compact factual chain may capture only part of the evidence requirement. Neighborhood expansion and graph diffusion can provide broader contextual coverage, offering a plausible explanation for their competitiveness in this setting.
The results in Table 1 reflect this distinction: ECG-RAG achieves the highest EM on all three multi-hop benchmarks and the highest F1 on 2WikiMultiHopQA and HotpotQA. It also achieves the highest F1 on Mix, while IGMiRAG and Cog-RAG perform slightly better on Pathology. As discussed in Section 4.2, the gains are less consistent on explanatory QA. The method therefore shows its clearest benefits on multi-hop evidence recovery while remaining competitive on explanatory tasks. CCC provides a complementary diagnostic alongside ER and end-to-end QA scores.
Response to Reviewer Tp3R (Part 1/8)
We thank the reviewer for recognizing the motivation and design of ECG-RAG and for the questions about evidence completeness, component contributions, and reproducibility.
Q1&W1. Importance of complete evidence chains for downstream QA.
We address this concern with two complementary context-intervention experiments on 2WikiMultiHopQA. The first examines whether average Evidence Recall (ER) alone captures the evidence needed for accurate answers. The second measures the effect of removing supporting sentences from ECG-RAG's actual generation contexts.
Controlled evidence-coverage analysis. We evaluated all 234 eligible four-supporting-fact questions in the original 2WikiMultiHopQA evaluation set. These are bridge-comparison questions. We constructed short contexts from gold supporting sentences, replacing omitted evidence with length-matched distractor sentences. Within each question, context lengths differ by at most one token, while the question, generation instructions, and generator settings remain unchanged.
1. Separating average evidence recall from complete-chain coverage. We compared three evidence-coverage conditions:
Results average two complementary assignments, so every question receives complete evidence in one assignment and partial evidence in the other for Conditions B and C. This balances the allocation of complete evidence across questions.
At the same average ER of 75%, increasing CCC from 0% to 50% improves EM by 27.56 percentage points and F1 by 27.42 percentage points. The respective 95% confidence intervals are [21.37, 33.55] and [21.30, 33.40], with Holm-adjusted p=0.00020 for both metrics. Conversely, at the same CCC of 50%, increasing average ER from 75% to 87.5% improves both EM and F1 by 6.62 points (pHolm=0.00100). These comparisons use 20,000 question-paired bootstrap resamples, two-sided tests, and Holm correction across the four metric-level comparisons.
These results show that identical average evidence recall can conceal substantial differences in answer support: providing partial evidence for every question is not equivalent to providing complete evidence for a subset. They also show that partial evidence remains useful, supporting the use of CCC alongside ER.