acceptodds
Under review as a conference paper at ICLR 2027

ECG-RAG: Evidence Chain-Guided Retrieval for Multi-Hop Question Answering

10 views

Abstract

Retrieval-Augmented Generation (RAG) is widely used to improve the factual grounding of Large Language Models (LLMs) by leveraging external knowledge. While effective for simple queries, standard RAG systems struggle with multi-hop questions where the necessary evidence is scattered across multiple passages. Recent graph-based methods capture relations among entities and text units to support broader context retrieval for complex reasoning tasks. However, existing dense and graph-based retrievers often optimize local relevance or neighborhood coverage, without explicitly ensuring that the retrieved evidence forms a complete support chain from the question to the answer. In this paper, we propose Evidence Chain-Guided Retrieval-Augmented Generation (ECG-RAG), a retrieval framework that explicitly targets evidence-chain completeness for multi-hop RAG. To support this goal, ECG-RAG adopts a hierarchical hypergraph that aligns atomic entities, pairwise relations, high-level multi-entity relations, and source passages in a unified retrieval space. At inference time, ECG-RAG parses each query into an evidence-chain goal and activates anchor nodes in the pairwise-relation layer. It then uses ant-inspired exploration to dynamically prioritize relation transitions that advance this goal, thereby deriving a compact query-specific local relation graph for subsequent path search. A-style search is then applied over this local graph to assemble evidence paths that cover the target chain. The selected paths are projected back to grounded source passages and organized into path-structured evidence context for answer generation. Extensive experiments on six QA benchmarks show that ECG-RAG achieves the best overall QA performance, reaching 60.4 EM and 66.6 F1 with average gains of 2.1 EM and 0.7 F1 over the strongest baseline. Retrieval diagnostics on 2WikiMultiHopQA show 93.6% Evidence Recall, with all annotated supporting evidence covered in the final generation context for 85.0% of the questions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Retrieval-Augmented Generation, Multi-Hop Question Answering, Evidence Chains, Graph Retrieval, Path Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Tp3ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author proposed a retrieval framework that explicitly targets evidence-chain completeness for multi-hop RAG. Specifically, the author adopts a hierarchical hypergraph, aligning everything in a unified retrieval space. At inference time, the proposed method parses the query into an evidence-chain goal and derives a compact, query-specific local relation graph for subsequent path search. The proposed method achieved the best overall performance on some benchmarks.

Critical strengths
  1. The paper revealed and investigated an interesting observation that there is a gap between evidence recall and complete evidence-chain coverage.
  2. The method design is complete and intuitive for solving the problem.
  3. The experiments are comprehensive, covering different types of Table-QA benchmarks, and multiple analyses are provided.
Critical weaknesses
  1. My main concern is the importance of the complete evidence chain, since the final performance improvement is slight and ECG-RAG does not achieve the best results on every metric. I was wondering how much an incomplete evidence chain affects QA accuracy or downstream tasks. The authors show that even with high recall, complete evidence-chain coverage is low, and in the main results, ECG-RAG sometimes achieves nearly identical accuracy to baselines.
  2. The method consists of multiple parts and the ablation is coarse-grained and only on one dataset; the necessity of some designs is not well justified. For example, are both Ant-inspired Exploration and A-style Search necessary? Can we replace them with a simpler method?
  3. If the evidence-chain goal generated by an LLM is itself incomplete or incorrect, can subsequent searches still retrieve the correct evidence?
  4. The method consists of multiple hyperparameters, but some key hyperparameter values are missing, which affects the reproducibility and clarity of the method. For example, the values of alpha, beta, and others are not specified.
Questions for authors

Please see the weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Tp3R (Part 1/8)

We thank the reviewer for recognizing the motivation and design of ECG-RAG and for the questions about evidence completeness, component contributions, and reproducibility.

Q1&W1. Importance of complete evidence chains for downstream QA.

We address this concern with two complementary context-intervention experiments on 2WikiMultiHopQA. The first examines whether average Evidence Recall (ER) alone captures the evidence needed for accurate answers. The second measures the effect of removing supporting sentences from ECG-RAG's actual generation contexts.

Controlled evidence-coverage analysis. We evaluated all 234 eligible four-supporting-fact questions in the original 2WikiMultiHopQA evaluation set. These are bridge-comparison questions. We constructed short contexts from gold supporting sentences, replacing omitted evidence with length-matched distractor sentences. Within each question, context lengths differ by at most one token, while the question, generation instructions, and generator settings remain unchanged.

1. Separating average evidence recall from complete-chain coverage. We compared three evidence-coverage conditions:

ConditionAverage ER (%)CCC (%)EM (%)F1 (%)
A: Every question receives 3/4 supporting facts75.00.025.2125.50
B: Half receive 4/4; half receive 2/475.050.052.7852.92
C: Half receive 4/4; half receive 3/487.550.059.4059.54

Results average two complementary assignments, so every question receives complete evidence in one assignment and partial evidence in the other for Conditions B and C. This balances the allocation of complete evidence across questions.

At the same average ER of 75%, increasing CCC from 0% to 50% improves EM by 27.56 percentage points and F1 by 27.42 percentage points. The respective 95% confidence intervals are [21.37, 33.55] and [21.30, 33.40], with Holm-adjusted p=0.00020p=0.00020 for both metrics. Conversely, at the same CCC of 50%, increasing average ER from 75% to 87.5% improves both EM and F1 by 6.62 points (pHolm=0.00100p_{\mathrm{Holm}}=0.00100). These comparisons use 20,000 question-paired bootstrap resamples, two-sided tests, and Holm correction across the four metric-level comparisons.

These results show that identical average evidence recall can conceal substantial differences in answer support: providing partial evidence for every question is not equivalent to providing complete evidence for a subset. They also show that partial evidence remains useful, supporting the use of CCC alongside ER.

AuthorsOpenReview ↗

Response to Reviewer Tp3R (Part 2/8)

Q1&W1. Importance of complete evidence chains for downstream QA.

Controlled evidence-coverage analysis.

2. Fine-grained analysis of evidence recall and completeness. Using the existing answer scores for the same 234 questions, we formed weighted mixtures of the zero-to-four supporting-fact retention levels to examine average ER and CCC from 0% to 75% in 15-percentage-point increments. For each target pair, the weight on the 4/4 condition equals CCC/100; the remaining weight is distributed between adjacent incomplete-retention levels to match the target average ER. The same weights are applied to every question.

EM (%).

Average ER (%) / CCC (%)01530456075
02.14
153.9315.85
306.5017.6529.57
4510.6020.7931.3743.29
6017.2625.5335.0945.2857.01
7525.2133.4841.7550.0259.5770.73

F1 (%).

Average ER (%) / CCC (%)01530456075
02.42
154.2216.10
306.7817.8929.77
4510.8821.0331.5743.45
6017.5525.7835.2845.4357.12
7525.5033.7341.9550.1859.6970.80

Rows indicate average ER and columns indicate CCC; blank cells represent infeasible combinations rather than zero scores.

Vertical comparisons hold CCC fixed and show that increasing ER improves both EM and F1, confirming the value of partial evidence. Horizontal comparisons hold average ER fixed and show the benefit of increasing complete-chain coverage by changing how evidence is distributed. Under this weighting scheme, adjacent 15-percentage-point increases in CCC at fixed average ER yield larger EM and F1 gains than adjacent 15-percentage-point increases in ER at fixed CCC throughout the displayed grid. These patterns show that, for these multi-hop questions and controlled contexts, evidence completeness provides important information beyond average recall, supporting CCC as a complementary diagnostic alongside ER.

3. QA performance across all evidence-retention levels. We further examined the full range from zero to four retained supporting facts on the same 234 questions:

Supporting facts retained per questionAverage ER (%)CCC (%)EM (%)F1 (%)
0/40.00.02.142.42
1/425.00.05.135.41
2/450.00.011.9712.25
3/475.00.025.2125.50
4/4100.0100.093.5993.59

Evidence subsets follow the same nested retention order. The 0/4 condition contains matched distractors rather than an empty context.

This breakdown reveals two complementary patterns. First, additional partial evidence improves QA even when CCC remains zero: moving from 2/4 to 3/4 supporting facts increases both EM and F1 by 13.25 points. Second, completing the support set produces a substantially larger gain: moving from 3/4 to 4/4 increases EM by 68.38 points and F1 by 68.09 points. Both transitions add one supporting fact and increase ER by 25 percentage points.

Together, the fixed-ER comparison and the retention-level breakdown show why evidence completeness matters for these multi-hop questions: partial evidence provides useful information, while recovering the remaining supporting fact can enable an answer that the incomplete context does not support.

AuthorsOpenReview ↗

Response to Reviewer Tp3R (Part 3/8)

Q1&W1. Importance of complete evidence chains for downstream QA.

Controlled evidence-coverage analysis.

4. Supporting-sentence deletion in actual generation contexts. We also analyzed 850 questions whose original ECG-RAG contexts covered all annotated supporting sentences. We compared deleting a selected supporting sentence and its exact duplicates with deleting matched non-gold sentence spans. Other context content and generation instructions were preserved, with final context lengths constrained to differ by at most 32 tokens and 0.5% of the original length.

ConditionER (%)CCC (%)EM (%)F1 (%)
Matched non-gold sentence deletion100.00100.0064.9472.86
Supporting-sentence deletion55.621.5351.0656.66

Paired interventions were successfully constructed for 837 questions and both conditions were regenerated. The remaining 13 questions retain identical historical predictions in both conditions and contribute zero paired effect; all 850 questions are included in the analysis.

Deleting supporting sentences reduces EM by 13.88 points and F1 by 16.20 points relative to the matched deletion control. The respective 95% confidence intervals are [11.18, 16.59] and [13.54, 18.93], with Holm-adjusted p=0.00010p=0.00010 for both metrics.

Both analyses use 20,000 question-paired bootstrap resamples, two-sided tests, and Holm correction within each analysis. Together, these interventions show that missing supporting evidence can substantially reduce downstream QA accuracy, including in ECG-RAG’s actual generation contexts. These findings provide empirical support for evaluating evidence completeness alongside average recall in multi-hop QA.

System comparison under a common context budget. We also conducted an additional comparison on 1,000 2WikiMultiHopQA questions with both methods subject to a 5,000-token final-context budget. ECG-RAG uses compact packing that removes redundant instructions and retrieval diagnostics while retaining relation-path content and packing supporting text within the budget.

MethodContext budgetAverage context tokensEMF1
IGMiRAG5,0004,740.5357.3066.12
ECG-RAG5,0004,806.5459.2067.46

ECG-RAG improves EM/F1 by 1.90/1.34 percentage points, while the actual average context lengths differ by 66.01 tokens, approximately 1.4%. This comparison connects the evidence-availability analyses above to end-to-end system performance under a shared context budget. The intervention experiments quantify sensitivity to missing evidence; this experiment measures the QA difference between retrieval systems.

Why the gains differ across task types. ECG-RAG primarily targets multi-hop QA, where answering requires connecting distributed facts through intermediate entities or relations. Missing one necessary link can leave a question unsupported even when other retrieved passages are relevant. Explanatory QA instead requires broader information synthesis and coherent descriptions, for which a single compact factual chain may capture only part of the evidence requirement. Neighborhood expansion and graph diffusion can provide broader contextual coverage, offering a plausible explanation for their competitiveness in this setting.

The results in Table 1 reflect this distinction: ECG-RAG achieves the highest EM on all three multi-hop benchmarks and the highest F1 on 2WikiMultiHopQA and HotpotQA. It also achieves the highest F1 on Mix, while IGMiRAG and Cog-RAG perform slightly better on Pathology. As discussed in Section 4.2, the gains are less consistent on explanatory QA. The method therefore shows its clearest benefits on multi-hop evidence recovery while remaining competitive on explanatory tasks. CCC provides a complementary diagnostic alongside ER and end-to-end QA scores.

Reviewer_cFj1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ECG-RAG, a retrieval framework aimed at recovering complete supporting evidence for multi-hop QA. It combines a hierarchical hypergraph with question-specific evidence goals, ant-inspired exploration, and heuristic path search. Experiments on QA benchmarks report improved average performance.

Critical strengths
  1. The paper usefully distinguishes evidence recall from complete evidence coverage. The analysis shows that a retriever can recover many relevant facts while still missing part of the evidence needed to answer a question (Figure 1). This gives a more informative view of retrieval failures in multi-hop QA.
  2. The retrieval evaluation provides evidence for the proposed approach. At top-15 on 2WikiMultiHopQA, ECG-RAG improves complete coverage over HippoRAG2 despite similar evidence recall.
Critical weaknesses
  1. Section 3.2 does not explain how contextual support and relation-argument compatibility are computed or how their thresholds are chosen. These scores determine which transitions are explored and which goal steps count as covered. The reported coverage gains are useful, but these definitions are still needed to understand and reproduce the retrieval procedure.

  2. The ablations show gains over static expansion and ranked selection, but do not establish whether both ant exploration and heuristic search are needed. A simpler goal-guided search with comparable resources would help answer that question. Separately, the matched-passage diagnostics support better evidence coverage, but do not isolate its contribution to QA performance when final-context budgets differ from IGMiRAG.

Questions for authors
  1. How are contextual support and relation-argument compatibility computed, and how are their thresholds chosen? Please also clarify how transition preferences are normalized.
  2. Are candidate counts, expansion budgets, and final-context lengths comparable across the search ablations? Are results available for a simpler goal-guided search under those settings?
  3. Do you have an end-to-end comparison with IGMiRAG under a common final-context budget?
7 comments
AuthorsOpenReview ↗

Response to Reviewer cFj1 (Part 1/5)

We thank the reviewer for highlighting the distinction between evidence recall and complete evidence coverage, and for the suggestions on implementation clarity and controlled comparisons.

Q1&W1. Contextual support, relation–argument compatibility, thresholds, and transition normalization.

Contextual support. In Section 3.2, Eq. (3), uq(lj)u_q(l_j) measures local contextual support for a candidate relation. It is computed deterministically from query-analysis outputs and existing index information, without additional LLM calls for individual candidates.

Text matching. Let Wq\mathcal W_q denote the query-term set extracted from the question and query-analysis outputs, and let Φq\Phi_q denote the corresponding normalized query phrases. For text xx, Tok⁡(x)\operatorname{Tok}(x) contains its normalized terms after stop-word and single-character filtering. Word and phrase coverage are computed as

\displaystyle c_{\mathrm{word}}(q,x)=\min\left\\{1,\frac{|\mathcal W_q\cap\operatorname{Tok}(x)|}{\max(1,\min(|\mathcal W_q|,12))}\right\\},

Here, 12 is the fixed normalization cap, and the word-coverage score is clipped to [0,1][0,1].

cphrase(q,x)=∑ϕ∈Φq1[ϕ occurs in norm⁡(x)]max⁡(1,∣Φq∣).\displaystyle c_{\mathrm{phrase}}(q,x)=\frac{\sum_{\phi\in\Phi_q}\mathbf{1}[\phi\text{ occurs in }\operatorname{norm}(x)]}{\max(1,|\Phi_q|)}.

Normalization lowercases text, replaces non-alphanumeric characters with spaces, and collapses whitespace. The shared text-matching function is

m(q,x)=max⁡cword(q,x),cphrase(q,x).\displaystyle m(q,x)=\max\\{c_{\mathrm{word}}(q,x),c_{\mathrm{phrase}}(q,x)\\}.

Support signals. We compute six signals bounded in [0,1][0,1]. RR is the word coverage cwordc_{\mathrm{word}} of the candidate relation's metadata; EE and HH measure matching support from participating entities and associated high-level relations, respectively; TT measures matching support from associated source chunks; and AA measures support from descriptions and attributes of the candidate relation, its participating entities, and its associated high-level relations. Node metadata comprises the node text, description, and attributes.

For E,H,T,AE,H,T,A, we apply m(q,x)m(q,x) to the corresponding texts and take the maximum score over associated items. If a participating entity is an activated entity anchor, E=1E=1; if an associated high-level relation is activated, H=1H=1. The sixth signal, PactP_{\mathrm{act}}, is the maximum existing activation score among the candidate relation and its associated entities, high-level relations, and source chunks, clipped to [0,1][0,1]. Missing signals receive zero.

Aggregation. These signals form three support views:

starget=max⁡(R,E,Pact),sstructure=max⁡(E,H),sgrounding=max⁡(T,A).\displaystyle s_{\mathrm{target}}=\max(R,E,P_{\mathrm{act}}),\qquad s_{\mathrm{structure}}=\max(E,H),\qquad s_{\mathrm{grounding}}=\max(T,A).

The contextual-support score is

uq(lj)=starget+sstructure+sgrounding3⋅n+3⋅(1−ρ(lj)),\displaystyle u_q(l_j)=\frac{s_{\mathrm{target}}+s_{\mathrm{structure}}+s_{\mathrm{grounding}}}{3}\cdot\frac{n_+}{3}\cdot\bigl(1-\rho(l_j)\bigr),

where n+n_+ counts the nonzero support views. This factor favors candidates supported across multiple views. The hub penalty downweights candidates connected through high-degree entities:

\displaystyle \rho(l_j)=\begin{cases}\displaystyle \min\left\\{1,\frac{\log(1+d(l_j))}{\log(1+M_{\mathrm{nbr}})}\right\\}, & d(l_j)>1,\\\\ 0, & \text{otherwise}.\end{cases}

Here, d(lj)d(l_j) is the maximum relation degree among the candidate's participating entities in the local graph, and MnbrM_{\mathrm{nbr}} is the configured neighborhood bound.

Thus, uqu_q provides local contextual support, while aqa_q measures progress toward eligible factual steps under dependency and slot-binding constraints (Section 3.2, Eq. (4)). They jointly guide exploration through ηq=1−(1−uq)(1−aq)\eta_q=1-(1-u_q)(1-a_q).

AuthorsOpenReview ↗

Response to Reviewer cFj1 (Part 2/5)

Q1&W1. Contextual support, relation–argument compatibility, thresholds, and transition normalization.

Relation–argument compatibility and thresholds. A factual step becomes eligible for matching once its prerequisites have been covered. For each source-grounded candidate fact, matching checks the predicate and the subject–object arguments jointly. Predicate matching requires equality after text normalization and resolution through any supplied predicate-alias mapping. Subject and object arguments are checked in their respective positions to preserve relation direction.

Explicit entities are resolved to canonical identifiers through exact names or supplied aliases. Argument matching distinguishes three cases: an explicit argument must equal the corresponding candidate value; an already-bound slot must agree with its value in BB; and an unbound slot can receive a grounded entity or literal value satisfying its type constraint.

In the exact/alias configuration described here, compatibility is binary:

σq(lj,gt;B)={1,a grounded candidate fact jointly satisfies predicate matchingand argument consistency under B,0,otherwise.\displaystyle \sigma_q(l_j,g_t;B)=\begin{cases}1, & \text{a grounded candidate fact jointly satisfies predicate matching}\\\\ & \text{and argument consistency under }B,\\\\ 0, & \text{otherwise}.\end{cases}

A successful match produces a consistent binding extension B′B' that preserves existing assignments. The extension is carried forward when the ant selects the transition or the path search creates the corresponding successor state. Each path maintains its own bindings, and alternative consistent assignments are represented separately.

We use δ=1\delta=1, so an eligible factual step is counted as covered when both matching conditions hold. Since σq∈0,1\sigma_q\in\\{0,1\\}, any δ∈(0,1]\delta\in(0,1] yields the same acceptance decisions in this configuration.

Transition normalization. After exploration, accumulated transition preferences are normalized by their query-wide maximum:

τq(e)=Δτq(e)max⁡e′∈EqLΔτq(e′).\displaystyle \tau_q(e)=\frac{\Delta\tau_q(e)}{\max_{e'\in\mathcal{E}_q^L}\Delta\tau_q(e')}.

If the maximum is zero, all normalized preferences are set to zero. This maps preferences to [0,1][0,1] while preserving their relative magnitudes, providing the path-search edge cost 1−τq(e)1-\tau_q(e). During exploration, τ~q(e)=max⁡(Δτq(e),0.01)\tilde{\tau}_q(e)=\max(\Delta\tau_q(e),0.01) supplies a fixed exploration floor, and sampling weights are normalized over the current neighbors. This floor is distinct from the compatibility threshold.

Related sampling parameters. In the exploration sampling rule (Section 3.2, Eq. (5)), we use α=β=1\alpha=\beta=1, so transition preferences and the heuristic enter the sampling weight to the first power. The exploration floor is ϵ=0.01\epsilon=0.01, while the factual-step acceptance threshold is δ=1\delta=1 in the binary exact/alias configuration described above. The exponents control sampling weights, the floor permits exploration of unreinforced transitions, and the compatibility threshold determines whether an eligible step counts as covered.

AuthorsOpenReview ↗

Response to Reviewer cFj1 (Part 3/5)

Q2&W2. Comparability across search ablations and simpler goal-guided alternatives.

Experimental settings. In the single-component search ablations, we remove only the designated component while keeping the other configuration parameters unchanged, including candidate-retrieval limits, applicable search expansion budgets, and the final-context token budget. Actual candidate counts, expansion counts, and context lengths can differ because they are outputs of each retrieval pipeline; the configured limits remain fixed.

Average final-context length (tokens per query).

Configuration2WikiMuSiQue
ECG-RAG (Full)8,979.319,214.46
w/o Relation Exploration7,780.768,783.06
w/o A∗A^\ast-style Path Retrieval6,438.206,689.73
w/o Path-Structured Context7,417.977,853.92

Simpler alternatives. The ablations in Section 4.4 evaluate two simplified configurations that retain goal guidance:

  • Without relation exploration: static anchor expansion replaces ant-inspired exploration, while A∗A^\ast-style goal-guided path retrieval is retained.
  • Without goal-guided path retrieval: ranked-candidate selection replaces A∗A^\ast-style path assembly, while ant-inspired exploration continues to use the evidence-chain goal.

The first variant retains goal guidance during path assembly, whereas the second retains it during exploration. Removing either stage reduces QA performance and Complete Chain Coverage in the 2WikiMultiHopQA ablation study. These results compare the full design with simpler pipelines, both simplified goal-guided pipelines achieve lower EM, F1, and Complete Chain Coverage than full ECG-RAG..

A further simplification would be goal-guided greedy traversal, which selects the next relation by immediate factual-step coverage gain and uses contextual support to break ties. This would remove the separate exploration stage and the comparison of multiple partial paths, but could overlook necessary intermediate relations that provide little immediate coverage gain. This alternative has not been evaluated in our current experiments. We thank the reviewer for this suggestion and will investigate such simplifications in future work, aiming to reduce search complexity while preserving evidence-chain completeness.

Why combine the two stages? Our motivation is to recover complete support chains within a constrained search space. Ant-inspired exploration discovers useful relation transitions, estimates query-specific transition preferences, and constructs a compact local relation graph, thereby narrowing the space for subsequent path search. A∗A^\ast-style search then uses this graph and its transition preferences to compare partial paths according to accumulated path cost and factual-step coverage, and to assemble candidate evidence chains. The two stages play complementary roles: exploration identifies and prioritizes useful transitions, while path search compares their combinations for goal-aligned evidence assembly.

Reviewer_kS9mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ECG-RAG aims to retrieve complete evidence chains for multi-hop QA. It uses the H3G index from IGMiRAG and first breaks a question into factual steps to be resolved. It then combines ant-inspired exploration with an A*-style search to find supporting relation paths and organize the corresponding passages for answer generation. On six benchmarks, it achieves average scores of 60.4 EM and 66.6 F1, compared with 58.3 EM and 65.9 F1 for IGMiRAG. The paper also includes evidence-coverage analysis and ablation experiments.

Critical strengths

The paper focuses on a useful problem in multi-hop QA: retrieving relevant passages does not always provide all the evidence needed to answer a question.

Critical weaknesses

Compared with IGMiRAG, ECG-RAG improves the average scores by 2.1 EM and 0.7 F1. However, the paper does not compare runtime under the same conditions, making it unclear whether these gains justify the added complexity of the retrieval pipeline.

Questions for authors

Q1. How is u_q (l_j) calculated? This is not explained in the paper. Please provide the details.

Q2. In Section 4.4, removing relation exploration or path-structured context leads to relatively small drops in EM and F1. Are these differences statistically significant? How much additional runtime does each component introduce?

Q3. In Table 1, the average improvement over IGMiRAG is modest. Please compare ECG-RAG’s runtime with other methods to show whether these gains come at a higher computational cost.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kS9m (Part 1/3)

Thank you for recognizing the importance of complete evidence-chain retrieval for multi-hop QA. We address the scoring details, ablation significance, and runtime comparisons below.

Q1. Calculation of uq(lj)u_q(l_j).

In Section 3.2, Eq. (3), uq(lj)u_q(l_j) measures local contextual support for a candidate relation. It is computed deterministically from query-analysis outputs and existing index information, without additional LLM calls for individual candidates.

Text matching. Let Wq\mathcal W_q denote the query-term set extracted from the question and query-analysis outputs, and let Φq\Phi_q denote the corresponding normalized query phrases. For text xx, Tok⁡(x)\operatorname{Tok}(x) contains its normalized terms after stop-word and single-character filtering. Word and phrase coverage are computed as

\displaystyle c_{\mathrm{word}}(q,x)=\min\left\\{1,\frac{|\mathcal W_q\cap\operatorname{Tok}(x)|}{\max(1,\min(|\mathcal W_q|,12))}\right\\},

Here, 12 is the fixed normalization cap, and the word-coverage score is clipped to [0,1][0,1].

$\displaystyle c_{\mathrm{phrase}}(q,x)= \frac{\sum_{\phi\in\Phi_q}\mathbf{1}[\phi\text{ occurs in }\operatorname{norm}(x)]}{\max(1,|\Phi_q|)}. $

Normalization lowercases text, replaces non-alphanumeric characters with spaces, and collapses whitespace. The shared text-matching function is

$\displaystyle m(q,x)=\max\{c_{\mathrm{word}}(q,x),c_{\mathrm{phrase}}(q,x)\}. $

Support signals. We compute six signals bounded in [0,1][0,1]. RR is the word coverage cwordc_{\mathrm{word}} of the candidate relation's metadata; EE and HH measure matching support from participating entities and associated high-level relations, respectively; TT measures matching support from associated source chunks; and AA measures support from descriptions and attributes of the candidate relation, its participating entities, and its associated high-level relations. Node metadata comprises the node text, description, and attributes.

For E,H,T,AE,H,T,A, we apply m(q,x)m(q,x) to the corresponding texts and take the maximum score over associated items. If a participating entity is an activated entity anchor, E=1E=1; if an associated high-level relation is activated, H=1H=1. The sixth signal, PactP_{\mathrm{act}}, is the maximum existing activation score among the candidate relation and its associated entities, high-level relations, and source chunks, clipped to [0,1][0,1]. Missing signals receive zero.

Aggregation. These signals form three support views:

$\displaystyle s_{\mathrm{target}}=\max(R,E,P_{\mathrm{act}}),\qquad s_{\mathrm{structure}}=\max(E,H),\qquad s_{\mathrm{grounding}}=\max(T,A). $

The contextual-support score is

$\displaystyle u_q(l_j)= \frac{s_{\mathrm{target}}+s_{\mathrm{structure}}+s_{\mathrm{grounding}}}{3} \cdot\frac{n_+}{3}\cdot\bigl(1-\rho(l_j)\bigr), $

where n+n_+ counts the nonzero support views. This factor favors candidates supported across multiple views. The hub penalty downweights candidates connected through high-degree entities:

$\displaystyle \rho(l_j)= \begin{cases} \displaystyle \min\left\{1,\frac{\log(1+d(l_j))}{\log(1+M_{\mathrm{nbr}})}\right\}, & d(l_j)>1,\\ 0, & \text{otherwise}. \end{cases} $

Here, d(lj)d(l_j) is the maximum relation degree among the candidate's participating entities in the local graph, and MnbrM_{\mathrm{nbr}} is the configured neighborhood bound.

Thus, uqu_q provides local contextual support, while aqa_q measures progress toward eligible factual steps under dependency and slot-binding constraints (Section 3.2, Eq. (4)). They jointly guide exploration through ηq=1−(1−uq)(1−aq)\eta_q=1-(1-u_q)(1-a_q).

AuthorsOpenReview ↗

Response to Reviewer kS9m (Part 2/3)

Q2. Ablation significance and component-level runtime.

Thank you for asking about the statistical reliability and runtime cost of the individual components.

To complement the ablations in Section 4.4, Table 2, and Appendix E, Table 8, we conducted additional evaluations and exploratory significance tests on both datasets. We used 20,000 paired bootstrap resamples over 1,000 questions per dataset with two-sided tests. The following results correspond to these additional runs; Δ\Delta denotes the full model's improvement over the ablation in percentage points.

DatasetAblationΔ\DeltaEMRaw pp (EM)Δ\DeltaF1Raw pp (F1)
2Wikiw/o Relation Exploration+2.300.0406+1.590.0363
2Wikiw/o Goal-Guided Path Retrieval+4.800.0065+3.460.0048
2Wikiw/o Path-Structured Context+1.900.0540+0.900.0613
MuSiQuew/o Relation Exploration+3.000.0188+2.590.0290
MuSiQuew/o Goal-Guided Path Retrieval+5.200.00010+6.140.00005
MuSiQuew/o Path-Structured Context+1.400.0283+1.690.0457

For the two components highlighted by the reviewer, relation exploration improves both metrics on both datasets, with all four comparisons reaching nominal significance at p<0.05p<0.05. Path-structured context also yields positive differences on both datasets, reaching nominal significance on MuSiQue; its smaller differences on 2Wiki do not reach the 0.05 threshold. Goal-guided path retrieval exhibits the largest performance differences on both benchmarks.

These effects correspond to the roles of the two search stages: Ant-inspired exploration discovers relation transitions and estimates their query-specific preferences while constructing a compact local graph. A∗A^\ast-style retrieval then compares partial paths on that graph using accumulated path cost and factual-step coverage to assemble evidence chains.

Separately, profiling the full evaluation run on 1,000 2Wiki questions provides the following mean wall-clock times for the stages associated with the three ablated components:

Component / Profiled StagesMain OperationsMean Time (s/query)
Ant-inspired relation explorationCandidate relation scoring, transition sampling, and preference updates0.313
Goal-guided A∗A^\ast path retrievalSearch-state expansion, path comparison, and candidate-path selection0.128
Path-structured context: evidence packing and final assemblySource-evidence selection, path-information organization, and final context assembly0.273

The two core retrieval stages together take approximately 0.44 seconds per query. Evidence packing and final context assembly together take 0.273 seconds per query. This stage-level breakdown provides a runtime reference alongside the observed ablation effects.

AuthorsOpenReview ↗

Response to Reviewer kS9m (Part 3/3)

Q3&W1. Accuracy-runtime trade-off.

We compared runtime on 2WikiMultiHopQA under matching hardware, concurrency, and timing boundaries, with 1,000 evaluated questions per method. We report amortized time per query, calculated by dividing the recorded batch wall-clock time by the number of questions.

MethodAmortized Time (s/query)
Naive RAG0.374
LightRAG0.785
NodeRAG2.366
PathRAG4.366
Hyper-RAG0.564
Cog-RAG1.390
IGMiRAG0.894
ECG-RAG0.924

ECG-RAG takes 0.924 seconds per query, corresponding to the rounded 0.92-second value reported in Appendix C, Table 6. Compared with IGMiRAG's 0.894 seconds, this represents approximately 0.030 additional seconds per query, or a 3.4% increase.

On the same benchmark, ECG-RAG achieves 60.9 EM and 68.1 F1, compared with IGMiRAG's 57.5 EM and 67.4 F1, improving performance by 3.4 EM and 0.7 F1 percentage points (Table 1, main paper). Thus, on 2Wiki, these QA improvements are accompanied by a modest increase in amortized runtime relative to IGMiRAG. ECG-RAG also records lower amortized times than NodeRAG, PathRAG, and Cog-RAG in this comparison.Overall, ECG-RAG achieves the highest EM and F1 among the compared methods on 2WikiMultiHopQA while maintaining competitive amortized runtime, offering a favorable balance between QA accuracy and computational cost.

QA performance under a common context budget. In a separate additional experiment on 1,000 2WikiMultiHopQA questions, both methods use a 5,000-token final-context budget. ECG-RAG uses compact packing that removes redundant instructions and retrieval diagnostics while retaining relation-path content and packing supporting text within the budget.

MethodContext budgetAverage context tokensEMF1
IGMiRAG5,0004,740.5357.3066.12
ECG-RAG5,0004,806.5459.2067.46

ECG-RAG improves EM by 1.90 percentage points and F1 by 1.34 percentage points, with average context lengths differing by 66.01 tokens, approximately 1.4%. The original-run timings above and this budget-controlled experiment quantify two distinct aspects of the trade-off: runtime under our evaluation setup and QA performance under a shared final-context budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.