- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Fuzzy Evidence-Guided Reasoning (FEGR), a graph-based retrieval-augmented generation framework that aims to improve evidence reliability under limited context budgets. The key idea is to model four evidence properties (relevance, supportiveness, conflict, and uncertainty) using fuzzy memberships. These properties are aggregated into path-level reliability scores through a noncompensatory product t-norm. FEGR subsequently selects evidence paths using a two-stage greedy strategy that combines query-anchor coverage, reliability per token, and evidence overlap constraints. Experiments are conducted on five UltraDomain corpora against five representative RAG baselines. The authors report favorable pairwise win rates, particularly on Supportiveness and Conflict Avoidance.
Critical strengths
- The paper identifies an important limitation of graph-based RAG: structurally connected and semantically relevant evidence is not necessarily reliable. Explicitly considering weak support, conflict, uncertainty, and redundancy during context selection is a meaningful direction.
- The evaluation covers five domains and multiple established baselines. Beyond the main system comparisons, the authors investigate context-budget sensitivity, mixed-risk perturbations, and component contributions.
- Overall framework is coherent. FEGR integrates evidence-quality assessment with path-level scoring and budget-aware selection. This makes the framework relevant to practical context construction.
Critical weaknesses
- The contribution of evidence paths is unclear. The paper emphasizes selecting reliable evidence paths to improve RAG, but it does not directly show whether multi-hop paths are better than individual evidence units. A comparison under the same retrieval results and token budget would help clarify how much the graph structure contributes to the performance.
- The evaluation does not sufficiently validate factual reliability. The paper's main claim is to improve evidence reliability, but the evaluation relies entirely on LLM-based pairwise judgments. For example, if FEGR filters out conflicting evidence, the judge may consider the answer conflict-free, but this does not necessarily mean the answer is factually correct.
- The contribution of the evidence selection module is difficult to isolate. The main experiments compare complete RAG systems with different retrieval and graph construction methods. Therefore, it is unclear how much of the improvement comes from these differences rather than the proposed evidence selection method.
Questions for authors
- Since the paper allows individual evidence units to form paths, how much of FEGR's improvement comes from using relationships between evidence units, rather than simply scoring and selecting individual evidence units?
- Could the authors provide a more quantitative evaluation of whether FEGR actually improves the factual reliability of generated answers?
- Could the authors apply FEGR's evidence selection module to the retrieval results of other RAG systems to better distinguish the contributions of upstream retrieval and evidence selection?
3 comments
Response to Reviewer NBds (2/3)
Gold-reference Evaluation of Answer Correctness (W2 & Q2)
We thank the reviewer for the important suggestions regarding the evaluation of FEGR's factual reliability. Reducing conflicts in retrieved evidence does not necessarily imply that generated answers are more accurate. It is therefore necessary to directly assess agreement between final answers and gold answers beyond the original LLM Pairwise Evaluation. To this end, we have added a Gold-reference QA experiment on HotpotQA and 2WikiMultiHopQA to RQ1 of the revised manuscript, using Exact Match (EM) and Token F1 to quantitatively evaluate final-answer correctness without relying on the original LLM judge.
Specifically, we use 150 test questions from each benchmark and compare FEGR, LightRAG, HippoRAG2, and PathRAG under the same 6K Token Budget. We separately use GPT-4o-mini and Qwen3.5-9B as full-pipeline LLM backbones. With the same source corpora and test questions, each method independently performs its native index construction, retrieval, and answer generation. Gold Answers and Supporting-fact Annotations are not used during retrieval or answer generation; final answers are evaluated directly against the Gold Answers. The results are shown below.
Table NB2. Gold-reference QA evaluation on HotpotQA and 2WikiMultiHopQA (EM/F1, %).
| Backbone | Method | 2Wiki EM | 2Wiki F1 | Hotpot EM | Hotpot F1 |
|---|---|---|---|---|---|
| Qwen3.5-9B | FEGR | 49.33 | 59.20 | 49.33 | 67.10 |
| LightRAG | 40.67 | 49.42 | 38.00 | 53.75 | |
| HippoRAG2 | 46.00 | 54.54 | 45.33 | 61.70 | |
| PathRAG | 22.67 | 27.25 | 41.33 | 57.80 | |
| GPT-4o-mini | FEGR | 52.00 | 62.40 | 52.67 | 69.50 |
| LightRAG | 43.33 | 52.10 | 41.33 | 57.90 | |
| HippoRAG2 | 49.33 | 59.30 | 48.00 | 65.10 | |
| PathRAG | 25.33 | 31.75 | 43.33 | 60.50 |
Each benchmark contains 150 test queries per method and backbone. Bold values indicate the best results within each backbone and metric.
The results show that FEGR achieves the highest EM and F1 point estimates across both multi-hop QA benchmarks and both independent LLM backbones, consistently outperforming even the closest baseline, HippoRAG2. Together with the risk admission and robustness analyses in the original RQ3, these findings provide complementary evidence from two perspectives: evidence risk control and final-answer correctness. In particular, the new Gold-reference QA evaluation does not depend on the original five-dimensional LLM judge but directly uses external gold answers, further demonstrating that FEGR's end-to-end performance advantage extends beyond apparent answer consistency and conflict avoidance to measurable answer-correctness metrics.
Response to Reviewer NBds (3/3)
Attribution of Evidence Selection Contributions (W3 & Q3)
We thank the reviewer for the constructive suggestions regarding the attribution of contributions from FEGR's upstream retrieval and downstream evidence selection. FEGR is designed as a complete graph-based RAG system, with the primary research objective of constructing reliable and complementary generation contexts through Fuzzy Reliability Inference and Budgeted Evidence Selection after candidate evidence has been obtained through graph retrieval. To distinguish the end-to-end performance of the complete system from the independent contribution of the downstream selection mechanism, we conducted a dedicated controlled comparison in Appendix C: Controlled Evidence Selection under Shared Candidates. Unlike the full-system comparisons in RQ1, where each method retains its native indexing and retrieval pipeline, this experiment fixes FEGR's upstream retrieval outputs, Evidence Graph, Evidence Units, Structural Links, and Candidate Paths, while keeping the 6K Token Budget, Context Renderer, Answer Generator, and evaluation protocol identical. Only downstream Evidence Scoring and Selection Strategy are varied. By fixing the same upstream candidate evidence, this experiment directly examines the contribution of downstream scoring and selection mechanisms while eliminating additional variables arising from differences in retrievers and evidence organization. The results show that full FEGR achieves Overall win rates of 62.6%, 61.4%, 58.4%, and 58.8% against Relevance Greedy, MMR, Reliability Greedy, and Joint Greedy, respectively. Specifically, Reliability Greedy retains the same Path Reliability, Reliability Gate, and Overlap Control, while Joint Greedy combines reliability and coverage gain into a single-stage selection objective. Full FEGR still achieves Overall win rates of 58.4% and 58.8% against these two controlled strategies, respectively, supporting the incremental contribution of its two-stage budgeted selection mechanism. Together with the component ablations in RQ4, these results further indicate that FEGR's performance advantage cannot be explained solely by differences in upstream retrieval and graph construction; its reliability modeling and budgeted evidence selection also make independent contributions under fixed upstream conditions.
Response to Reviewer NBds (1/3)
We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments focus on the practical contribution of multi-unit evidence paths, the relationship between evidence reliability and the factual correctness of final answers, and the independent contribution of evidence selection relative to upstream retrieval. In response, we have further clarified FEGR's path-level modeling and budgeted selection mechanisms, and provided more direct empirical evidence and methodological explanations through newly added gold-reference QA experiments and existing controlled experiments under shared candidates. We address your comments individually below.
Contribution of Multi-unit Evidence Paths (W1 & Q1)
We thank the reviewer for the constructive suggestions regarding the independent contribution of multi-unit evidence paths. FEGR introduces Evidence Paths not merely to reorganize retrieved evidence, but to exploit structural relationships in the Evidence Graph, transforming individual Evidence Units into local evidence combinations that can be jointly evaluated and selected. Compared with directly scoring and admitting individual Evidence Units, path-level representation helps mitigate the omission of related evidence that may arise from local scoring. For example, certain intermediate relations or auxiliary fact units may have relatively low individual Relevance or Supportiveness, yet remain important when combined with other evidence to form a complete evidence chain. Through path-level Membership Aggregation, these units can be evaluated jointly for reliability as part of their associated paths and, when the admission conditions are satisfied, included together in the final Context. Meanwhile, FEGR uses Maximum aggregation for Conflict and Uncertainty to prevent stronger risks within a path from being diluted by other low-risk units. Retaining Singleton Paths also allows FEGR to handle queries that can be answered using individual evidence units without unnecessarily requiring multi-unit evidence combinations.
To directly quantify the incremental contribution of this design, we have added a controlled Path-level vs. Singleton-only ablation experiment to RQ4 of the revised manuscript. The experiment uses the same 100 queries across five domains and fixes the upstream retrieval outputs, underlying Evidence Units, Evidence Graph, four-dimensional Membership values, Reliability Gate, two-stage selection strategy, Context Renderer, Answer Generator, and 6K Token Budget, varying only the granularity of candidate evidence path organization. Full FEGR allows Evidence Paths of lengths 1–3, whereas Singleton-only restricts candidates to individual Evidence Units represented as paths of length 1. Both settings follow the same position-balanced, tie-aware pairwise evaluation protocol. The results are presented below.
Table NB1. Path-level vs. Singleton-only Evidence Selection under Shared Retrieved Evidence (FEGR win rate, %).
Values above 50% favor full FEGR. All results follow position-balanced, tie-aware pairwise evaluation.
The results show that, under identical upstream candidate evidence and context budgets, full FEGR achieves win rates above 50% across all five evaluation dimensions, with an Overall win rate of 54.2%. This controlled result provides direct empirical support for the incremental contribution of Multi-unit Evidence Paths over individual evidence unit selection, indicating that joint reliability assessment and collective evidence admission under structural relationship constraints can further improve final answer quality.