acceptodds
Under review as a conference paper at ICLR 2027

Reliable Evidence Path Selection in Graph-Based RAG via Fuzzy Evidence-Guided Reasoning

Abstract

Retrieval-augmented generation (RAG) grounds large language models in external knowledge, while graph-based RAG organizes retrieved information into entities, relations, and paths to support complex reasoning. However, graph connectivity and semantic relevance do not guarantee evidential reliability. Candidate paths may be relevant yet weakly supportive, conflicting, uncertain, or redundant. These conditions vary in degree and may coexist, while limited context budgets amplify the cost of admitting unreliable evidence. To address this problem, we propose Fuzzy Evidence-Guided Reasoning (FEGR), a graph-based RAG system for reliable evidence path selection. Specifically, FEGR models relevance, supportiveness, conflict, and uncertainty as fuzzy memberships, aggregates them at the path level through noncompensatory inference, and selects a compact set of complementary evidence paths under a token budget for answer generation. Across five domains, FEGR achieves Overall win rates above parity against all five representative RAG baselines under a common budget for retrieved context, with consistent gains in Supportiveness and Conflict Avoidance and lower admission of mixed evidence risks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, graph-based RAG, evidence path selection, evidence reliability, fuzzy inference, context budget

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NBdsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fuzzy Evidence-Guided Reasoning (FEGR), a graph-based retrieval-augmented generation framework that aims to improve evidence reliability under limited context budgets. The key idea is to model four evidence properties (relevance, supportiveness, conflict, and uncertainty) using fuzzy memberships. These properties are aggregated into path-level reliability scores through a noncompensatory product t-norm. FEGR subsequently selects evidence paths using a two-stage greedy strategy that combines query-anchor coverage, reliability per token, and evidence overlap constraints. Experiments are conducted on five UltraDomain corpora against five representative RAG baselines. The authors report favorable pairwise win rates, particularly on Supportiveness and Conflict Avoidance.

Critical strengths
  • The paper identifies an important limitation of graph-based RAG: structurally connected and semantically relevant evidence is not necessarily reliable. Explicitly considering weak support, conflict, uncertainty, and redundancy during context selection is a meaningful direction.
  • The evaluation covers five domains and multiple established baselines. Beyond the main system comparisons, the authors investigate context-budget sensitivity, mixed-risk perturbations, and component contributions.
  • Overall framework is coherent. FEGR integrates evidence-quality assessment with path-level scoring and budget-aware selection. This makes the framework relevant to practical context construction.
Critical weaknesses
  • The contribution of evidence paths is unclear. The paper emphasizes selecting reliable evidence paths to improve RAG, but it does not directly show whether multi-hop paths are better than individual evidence units. A comparison under the same retrieval results and token budget would help clarify how much the graph structure contributes to the performance.
  • The evaluation does not sufficiently validate factual reliability. The paper's main claim is to improve evidence reliability, but the evaluation relies entirely on LLM-based pairwise judgments. For example, if FEGR filters out conflicting evidence, the judge may consider the answer conflict-free, but this does not necessarily mean the answer is factually correct.
  • The contribution of the evidence selection module is difficult to isolate. The main experiments compare complete RAG systems with different retrieval and graph construction methods. Therefore, it is unclear how much of the improvement comes from these differences rather than the proposed evidence selection method.
Questions for authors
  1. Since the paper allows individual evidence units to form paths, how much of FEGR's improvement comes from using relationships between evidence units, rather than simply scoring and selecting individual evidence units?
  2. Could the authors provide a more quantitative evaluation of whether FEGR actually improves the factual reliability of generated answers?
  3. Could the authors apply FEGR's evidence selection module to the retrieval results of other RAG systems to better distinguish the contributions of upstream retrieval and evidence selection?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NBds (1/3)

We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments focus on the practical contribution of multi-unit evidence paths, the relationship between evidence reliability and the factual correctness of final answers, and the independent contribution of evidence selection relative to upstream retrieval. In response, we have further clarified FEGR's path-level modeling and budgeted selection mechanisms, and provided more direct empirical evidence and methodological explanations through newly added gold-reference QA experiments and existing controlled experiments under shared candidates. We address your comments individually below.

Contribution of Multi-unit Evidence Paths (W1 & Q1)

We thank the reviewer for the constructive suggestions regarding the independent contribution of multi-unit evidence paths. FEGR introduces Evidence Paths not merely to reorganize retrieved evidence, but to exploit structural relationships in the Evidence Graph, transforming individual Evidence Units into local evidence combinations that can be jointly evaluated and selected. Compared with directly scoring and admitting individual Evidence Units, path-level representation helps mitigate the omission of related evidence that may arise from local scoring. For example, certain intermediate relations or auxiliary fact units may have relatively low individual Relevance or Supportiveness, yet remain important when combined with other evidence to form a complete evidence chain. Through path-level Membership Aggregation, these units can be evaluated jointly for reliability as part of their associated paths and, when the admission conditions are satisfied, included together in the final Context. Meanwhile, FEGR uses Maximum aggregation for Conflict and Uncertainty to prevent stronger risks within a path from being diluted by other low-risk units. Retaining Singleton Paths also allows FEGR to handle queries that can be answered using individual evidence units without unnecessarily requiring multi-unit evidence combinations.

To directly quantify the incremental contribution of this design, we have added a controlled Path-level vs. Singleton-only ablation experiment to RQ4 of the revised manuscript. The experiment uses the same 100 queries across five domains and fixes the upstream retrieval outputs, underlying Evidence Units, Evidence Graph, four-dimensional Membership values, Reliability Gate, two-stage selection strategy, Context Renderer, Answer Generator, and 6K Token Budget, varying only the granularity of candidate evidence path organization. Full FEGR allows Evidence Paths of lengths 1–3, whereas Singleton-only restricts candidates to individual Evidence Units represented as paths of length 1. Both settings follow the same position-balanced, tie-aware pairwise evaluation protocol. The results are presented below.

Table NB1. Path-level vs. Singleton-only Evidence Selection under Shared Retrieved Evidence (FEGR win rate, %).

ComparisonRel.Sup.Conf.Log.Coh.Overall
Full FEGR vs. Singleton-only54.854.055.554.852.054.2

Values above 50% favor full FEGR. All results follow position-balanced, tie-aware pairwise evaluation.

The results show that, under identical upstream candidate evidence and context budgets, full FEGR achieves win rates above 50% across all five evaluation dimensions, with an Overall win rate of 54.2%. This controlled result provides direct empirical support for the incremental contribution of Multi-unit Evidence Paths over individual evidence unit selection, indicating that joint reliability assessment and collective evidence admission under structural relationship constraints can further improve final answer quality.

AuthorsOpenReview ↗

Response to Reviewer NBds (2/3)

Gold-reference Evaluation of Answer Correctness (W2 & Q2)

We thank the reviewer for the important suggestions regarding the evaluation of FEGR's factual reliability. Reducing conflicts in retrieved evidence does not necessarily imply that generated answers are more accurate. It is therefore necessary to directly assess agreement between final answers and gold answers beyond the original LLM Pairwise Evaluation. To this end, we have added a Gold-reference QA experiment on HotpotQA and 2WikiMultiHopQA to RQ1 of the revised manuscript, using Exact Match (EM) and Token F1 to quantitatively evaluate final-answer correctness without relying on the original LLM judge.

Specifically, we use 150 test questions from each benchmark and compare FEGR, LightRAG, HippoRAG2, and PathRAG under the same 6K Token Budget. We separately use GPT-4o-mini and Qwen3.5-9B as full-pipeline LLM backbones. With the same source corpora and test questions, each method independently performs its native index construction, retrieval, and answer generation. Gold Answers and Supporting-fact Annotations are not used during retrieval or answer generation; final answers are evaluated directly against the Gold Answers. The results are shown below.

Table NB2. Gold-reference QA evaluation on HotpotQA and 2WikiMultiHopQA (EM/F1, %).

BackboneMethod2Wiki EM2Wiki F1Hotpot EMHotpot F1
Qwen3.5-9BFEGR49.3359.2049.3367.10
LightRAG40.6749.4238.0053.75
HippoRAG246.0054.5445.3361.70
PathRAG22.6727.2541.3357.80
GPT-4o-miniFEGR52.0062.4052.6769.50
LightRAG43.3352.1041.3357.90
HippoRAG249.3359.3048.0065.10
PathRAG25.3331.7543.3360.50

Each benchmark contains 150 test queries per method and backbone. Bold values indicate the best results within each backbone and metric.

The results show that FEGR achieves the highest EM and F1 point estimates across both multi-hop QA benchmarks and both independent LLM backbones, consistently outperforming even the closest baseline, HippoRAG2. Together with the risk admission and robustness analyses in the original RQ3, these findings provide complementary evidence from two perspectives: evidence risk control and final-answer correctness. In particular, the new Gold-reference QA evaluation does not depend on the original five-dimensional LLM judge but directly uses external gold answers, further demonstrating that FEGR's end-to-end performance advantage extends beyond apparent answer consistency and conflict avoidance to measurable answer-correctness metrics.

AuthorsOpenReview ↗

Response to Reviewer NBds (3/3)

Attribution of Evidence Selection Contributions (W3 & Q3)

We thank the reviewer for the constructive suggestions regarding the attribution of contributions from FEGR's upstream retrieval and downstream evidence selection. FEGR is designed as a complete graph-based RAG system, with the primary research objective of constructing reliable and complementary generation contexts through Fuzzy Reliability Inference and Budgeted Evidence Selection after candidate evidence has been obtained through graph retrieval. To distinguish the end-to-end performance of the complete system from the independent contribution of the downstream selection mechanism, we conducted a dedicated controlled comparison in Appendix C: Controlled Evidence Selection under Shared Candidates. Unlike the full-system comparisons in RQ1, where each method retains its native indexing and retrieval pipeline, this experiment fixes FEGR's upstream retrieval outputs, Evidence Graph, Evidence Units, Structural Links, and Candidate Paths, while keeping the 6K Token Budget, Context Renderer, Answer Generator, and evaluation protocol identical. Only downstream Evidence Scoring and Selection Strategy are varied. By fixing the same upstream candidate evidence, this experiment directly examines the contribution of downstream scoring and selection mechanisms while eliminating additional variables arising from differences in retrievers and evidence organization. The results show that full FEGR achieves Overall win rates of 62.6%, 61.4%, 58.4%, and 58.8% against Relevance Greedy, MMR, Reliability Greedy, and Joint Greedy, respectively. Specifically, Reliability Greedy retains the same Path Reliability, Reliability Gate, and Overlap Control, while Joint Greedy combines reliability and coverage gain into a single-stage selection objective. Full FEGR still achieves Overall win rates of 58.4% and 58.8% against these two controlled strategies, respectively, supporting the incremental contribution of its two-stage budgeted selection mechanism. Together with the component ablations in RQ4, these results further indicate that FEGR's performance advantage cannot be explained solely by differences in upstream retrieval and graph construction; its reliability modeling and budgeted evidence selection also make independent contributions under fixed upstream conditions.

Reviewer_Fxv6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FEGR, a graph-based RAG method for selecting evidence under a limited context budget. For each retrieved evidence unit, it computes four scores (relevance, supportiveness, conflict and uncertainty) and combines them into a reliability score for each candidate path. It then greedily selects reliable, complementary paths that fit within the budget. In experiments on five domains, evaluated with pairwise LLM judgments, FEGR is preferred over a text-based baseline and several graph-based RAG baselines.

Critical strengths
  1. The paper addresses a well-defined and practically relevant question. Under a limited context budget, evidence that is relevant to the query may still be weakly supportive, conflicting, uncertain or redundant, and admitting it can push out more useful evidence.
  2. The authors show good familiarity with the graph-based RAG literature, and the related work covers the main lines of research in this area.
  3. Describing each piece of evidence along four interpretable dimensions, and combining them so that a serious risk cannot be offset by strong positive signals, is intuitive. The end-to-end example in the appendix makes these scores easy to inspect for concrete evidence units.
  4. The experiments go beyond a single comparison. They include a budget sweep, a stress test, ablations, and a control that fixes the candidate pool to isolate the selection step.
Critical weaknesses
  1. The main idea is to score each piece of retrieved evidence along four dimensions and use these scores to decide what enters the context. Ranking and filtering retrieved evidence has been widely studied, so the value of the paper rests largely on how these scores are designed and computed. However, although Section 3.3 states that the complete signal definitions and weights are given in Appendix A.1, the appendix describes them only at the level of signal families and rule types, without specifying how each signal is computed, the exact conflict rules, or the signal and rule weights. This makes it hard to see what distinguishes the approach from existing evidence selection methods, and hard to reproduce it.
  2. Several design choices are justified only briefly. The paper gives short intuitive reasons for the four dimensions and for the aggregation rules (geometric mean and max within a path, product across dimensions), but apart from comparing the product with an arithmetic mean in the ablation, it does not examine whether other dimensions or aggregation rules would work as well or better. Nor does it explain how the many fixed parameters and weights were selected, and there is no sensitivity analysis for these fixed parameters.
  3. The storyline could be tightened. The introduction emphasizes that graph-retrieved evidence can be weakly supportive, conflicting, uncertain or redundant, which may lead readers to expect improvements to graph construction, whereas the method filters evidence after retrieval. More importantly, the paper does not show how often these issues actually occur in the benchmarks, or how the gains depend on each of them. The stress test relies on perturbations that the authors constructed to mirror the risk types the method targets, and it reports only aggregate results rather than results for each risk type.
  4. The answer-quality evaluation relies entirely on pairwise judgments by GPT-4o-mini without reference answers, on a fairly small query set (40 per domain in the main comparison), and without significance tests. The judge is shown each system's retrieved context and asked to judge relative to it, and two of the main criteria, supportiveness and conflict avoidance, closely mirror what FEGR itself optimizes. Reference-based metrics or human evaluation would make the results much more convincing.
  5. The baselines are complete RAG systems (one text-based and four graph-based), and their relationship to FEGR is not clearly explained. Apart from simple selection variants in Appendix C (such as relevance-greedy and MMR), established methods that directly address evidence selection, such as rerankers, evidence filtering or compression, and conflict-aware RAG methods, are not compared. In addition, the paper does not report the computational overhead or scalability of FEGR itself, such as the per-query cost of building the evidence graph and computing the scores.
Questions for authors
  1. Could the authors describe exactly how the four scores are computed, including the concrete signals, the conflict rules and their weights?
  2. Since the evaluation queries have no reference answers, could the authors also evaluate FEGR on a benchmark with gold answers, or with human judgments on a subset of queries?
  3. How often do weakly supportive, conflicting, uncertain or redundant candidates actually occur in the benchmarks, and how do the gains vary across these types?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Fxv6 (1/5)

We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments address FEGR's scoring mechanism and its design rationale, methodological novelty, evaluation reliability, and computational efficiency, providing valuable guidance for strengthening the methodological justification and experimental analysis. In response, we have carefully examined these issues using the existing theoretical definitions, ablation and controlled experiments, and the newly added gold-reference QA evaluation. We have also supplemented the revised manuscript with the necessary technical details and result explanations. We address your comments individually below.

Membership Definition and Methodological Distinction (W1 & Q1)

Thank you for your specific suggestions regarding the computation of FEGR's four memberships and their distinction from conventional evidence selection methods. FEGR's four memberships are constructed from predefined operational evidence signals and fixed aggregation rules. Their signal definitions, normalization procedures, and aggregation mechanisms are described in Section 3.3 and Appendix A.1. Relevance, Supportiveness, and Uncertainty use fixed-weight aggregation of normalized evidence signals, with the available weights renormalized when certain signals are unavailable. Conflict, in contrast, uses local evidence comparison: only claims comparable in Subject, Attribute, and Scope are evaluated for conflict responses involving numerical values, dates, entity–value contradictions, explicit corrections, local negation, and mutually exclusive states. The maximum response is then aggregated into a continuous conflict-risk membership. To improve reproducibility, we have further specified the fixed signal and rule weights in Table 4 of the revised Appendix A.1, as shown below.

Table 4. Fixed operational signal and conflict-rule weights.

MembershipFixed Signal / Rule Weights
μrel\mu_{\mathrm{rel}}Retrieval confidence (0.45), typed-anchor coverage (0.35), structural correspondence (0.20)
μsup\mu_{\mathrm{sup}}Core-anchor coverage (0.25), predicate/relation coverage (0.20), requirement coverage (0.20), answer-bearing quality (0.20), assertion density (0.15)
μconf\mu_{\mathrm{conf}}Numeric contradiction (0.90), date contradiction (0.90), entity–value contradiction (0.85), explicit correction (0.80), local negation mismatch (0.65), mutually exclusive status (0.55)
μunc\mu_{\mathrm{unc}}Hedging (0.20), ambiguity (0.15), weak grounding (0.20), low retrieval confidence (0.15), content-form risk (0.15), incomplete scope (0.10), conflict unobservability (0.05)

FEGR's methodological contribution extends beyond assigning quality scores to individual evidence units and ranking them. As described in Sections 3.2–3.5, FEGR organizes structurally related Evidence Units into candidate Evidence Paths and jointly evaluates their supporting conditions and potential risks through path-level Membership Aggregation and noncompensatory Reliability Inference. Under the Reliability Gate, it then performs Coverage-first and Reliability-fill two-stage budgeted selection. Consequently, evidence admission depends not only on a candidate path's Reliability Score but also on its coverage gain, incremental Token Cost, and overlap with previously selected evidence. To further examine the contribution of multi-unit path organization, we added a controlled w/o Multi-unit Paths ablation to RQ4. Under the same upstream evidence and 6K Token Budget, full FEGR achieves an Overall win-rate point estimate of 54.2% against Singleton-only Selection. Together with the existing Membership Ablations and the Shared Candidates experiment in Appendix C, this result supports FEGR's methodological contribution in extending independent evidence scoring to structurally constrained joint reliability assessment and budgeted selection of complementary evidence.

AuthorsOpenReview ↗

Response to Reviewer Fxv6 (2/5)

Aggregation Design and Fixed Parameter Settings (W2)

We thank the reviewer for the constructive comments on FEGR's four membership dimensions, path-level reliability aggregation rules, and the rationale for its fixed parameter settings. The four dimensions characterize different aspects of evidence quality: Relevance measures the relationship between evidence and the query; Supportiveness measures the extent to which evidence supports the facts required to answer the query; Conflict captures local factual incompatibilities; and Uncertainty reflects uncertainty within the evidence itself. These dimensions serve distinct semantic roles: relevant evidence does not necessarily provide sufficient support, while factual conflict and informational uncertainty represent different reliability risks. Accordingly, FEGR assesses reliability through complementary evidence conditions rather than relying on a single aggregate quality score.

For within-path aggregation, Relevance and Supportiveness use the Geometric Mean (GM) to reduce the path-length effect associated with direct multiplication while retaining sensitivity to weaker evidence units. Conflict and Uncertainty use the Maximum (Max) operator so that stronger local risks are not diluted by other low-risk units. For cross-dimensional inference, FEGR employs multiplicative aggregation with fixed exponents, jointly incorporating positive supporting conditions and risk factors into path reliability computation. This avoids the additive compensation permitted by simple weighted summation. These aggregation choices reflect the distinct roles of evidence support and risk constraints, together forming FEGR's noncompensatory reliability inference mechanism.

The existing ablation results further support this design. In RQ4, full FEGR achieves a 59.4% Overall win rate against the Compensatory Mean variant, which replaces cross-dimensional multiplicative aggregation with the Arithmetic Mean. When Relevance, Supportiveness, Conflict, and Uncertainty are individually removed, full FEGR achieves Overall win rates of 56.0%, 58.1%, 59.2%, and 58.8%, respectively. These results support the complementary roles of the four evidence conditions and the effectiveness of noncompensatory aggregation relative to simple arithmetic averaging.

Regarding fixed parameters, FEGR uses predefined operational signal weights and reliability exponents. These settings were determined before the formal evaluation and kept consistent across the evaluated domains. The 200 formal evaluation queries were not used to tune these weights or rule parameters. The corresponding fixed weights, Conflict Rule Coefficients, and Reliability Exponents are explicitly disclosed in Appendix A.1–A.2 of the revised manuscript to ensure consistency between the methodological definitions and experimental configurations.

AuthorsOpenReview ↗

Response to Reviewer Fxv6 (3/5)

Research Scope and Evidence Risk Analysis (W3 & Q3)

We thank the reviewer for the constructive suggestions regarding FEGR's research scope and evidence risk analysis. We have adopted the suggestion concerning the research focus in the Introduction and further clarified in the revised manuscript that FEGR addresses reliable evidence assessment and context allocation after graph-based retrieval. Graph Connectivity and Query Relevance help identify relevant candidate evidence but do not directly guarantee sufficient support or reliability. Accordingly, FEGR focuses on improving the quality of the final generation context through path-level Reliability Inference and complementary evidence selection under a limited Token Budget.

Regarding the practical significance and experimental validation of evidence risks, we examine two complementary aspects: evaluation under standard task conditions and controlled risk analysis. RQ1 demonstrates FEGR's end-to-end performance advantages over five baselines across five domains under standard retrieval conditions without artificial risk injection. The revised manuscript further includes gold-reference QA experiments on HotpotQA and 2WikiMultiHopQA, where FEGR achieves the highest EM/F1 point estimates under both LLM backbones, providing additional evidence of its effectiveness on standard multi-hop question-answering tasks.

RQ3, through the Mixed-risk Stress Test, uses the injection proportion of four evidence risk types—Weakly Supportive, Conflicting, Uncertain, and Redundant—as a controlled variable to examine system robustness under increasing risk exposure. At a 50% mixed-risk injection level, FEGR achieves 90.0% Answer Robustness, compared with 86.0% for PathRAG, while its Risk Admission is 13.5%, compared with 23.8% for PathRAG. Furthermore, the RQ4 component ablations provide additional evidence regarding the contributions of different reliability modeling mechanisms. Together, these experiments support FEGR's research motivation and methodological effectiveness from three perspectives: performance on standard question-answering tasks, robustness under controlled evidence risks, and component-level contributions.

Reviewer_C4m9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that in graph-based RAG, being connected to the query graph and semantically relevant doesn't make a path reliable evidence. Under a fixed context budget, weakly supportive, conflicting, uncertain, or redundant paths can crowd out better ones. The proposed system, FEGR, scores each retrieved evidence unit with four [0,1] "memberships" (relevance, supportiveness, conflict, uncertainty). Three of these are fixed weighted averages of heuristic signals; conflict is a max over hand-written pairwise rules. Units are aggregated to paths (geometric mean for the positive conditions, max for the risks), and the four factors are multiplied into a reliability score R (product t-norm). A threshold gate and a two-phase greedy selector then pick paths: first anchor coverage per token within 35% of the budget, then reliability per token for the rest, with a Jaccard overlap filter on evidence identities. Evaluation uses 200 hand-validated queries over five UltraDomain corpora. A gpt-4o-mini judge compares answers pairwise against NaiveRAG, GraphRAG, LightRAG, PathRAG and HippoRAG2 under a 6K evidence cap, and the paper adds budget sweeps, a synthetic mixed-risk injection test, ablations, and a shared-candidate selector comparison (App. C).

Critical strengths
  1. The problem is well posed and practically relevant. Treating context construction as budgeted allocation by evidential value and risk, rather than by relevance or connectivity, is a sensible framing (Sec. 1, Sec. 3.1), and the noncompensatory argument for Eq. (4) is easy to follow. I also liked that budget accounting is strict: the final serialization is re-tokenized and an assertion fires instead of silent truncation (Sec. 3.6, App. B.2).

  2. The experimental design goes beyond a single headline table. There's a budget sweep from 3K to Full with a win/tie/loss breakdown (Fig. 3), a controlled injection test that reports both answer-level robustness and token-level risk admission (Fig. 4, Eqs. 16–18), membership and selection ablations (Table 2), and, most usefully, a shared-candidate control where only the selector changes (App. C, Table 10). That last one is the right experiment for isolating the selection contribution. Evaluation runs in both presentation orders with ties allowed, and the judge prompt is reproduced in full (Fig. 8).

  3. The authors are fairly careful about how they phrase results. They speak of "point estimates," say R is "not a calibrated probability," note that the narrowing at Full doesn't imply lower absolute quality, and say RQ3 measures within-system degradation rather than selector performance on an identical pool (Sec. 4.3–4.4). The numbers in the text match the tables: I recomputed all the domain-macro averages in Table 1 (57.3/73.3/58.5/65.1/56.6) and the per-dimension values quoted for LightRAG and HippoRAG2, and they check out.

Critical weaknesses
  1. All results come from a single gpt-4o-mini pairwise judge (also the generator), with no gold-answer metrics, human check, or significance tests on only 40 queries per domain. The claimed concentration of gains in Supportiveness and Conflict Avoidance also isn't visible in Table 1: Coherence is the top dimension against PathRAG (69.0) and HippoRAG2 (57.4), which looks more like a halo effect.

  2. The case study contradicts the stated gate. u1's memberships (0.383, 0.794, 0.650, 0.249) give R ≈ 0.080 under Eq. (4), below τ* = 0.10, yet singleton p1 is admitted first (Fig. 5, Table 6). Its 0.65 "conflict" comes from merely mixed reviews, which suggests the rule-based detector misfires and the product then suppresses the most useful evidence.

  3. The "fuzzy" scorer is a hand-tuned weighted heuristic whose signal weights, rule weights, and conflict-alignment procedure are never given, and the upstream index/retriever is unnamed (the rendered context matches LightRAG's format). With author-curated queries (App. B.1) and FEGR falling to about 49% against LightRAG at Full budget (Fig. 3), the "above parity against all baselines" claim is narrower than stated.

Questions for authors
  1. In the Fig. 5 / Table 6 case, how was p1 admitted when its singleton reliability under Eq. (4) is about 0.080 < τ* = 0.10? Is the implemented score different from Eq. (4), or is the gate applied differently in the coverage phase?
  2. What graph index and retriever does FEGR use upstream? Is it LightRAG's (the rendered sections suggest so)? If so, what does RQ1 vs. LightRAG look like when LightRAG's context is simply reranked by R under the same cap?
  3. Can you report results with gold-answer metrics (EM/F1 or LLM-judged correctness against references) on a standard multi-hop benchmark? Can you also report a stronger or different judge (e.g., GPT-4o or Claude), cross-order agreement rates, and a small human study on, say, 50 pairs?
  4. Please give all signal weights, rule weights λ_k, cue lists, and the conflict alignment procedure. How were they chosen, and were any of the 200 evaluation queries used during that tuning? What is the "J2 calibration variant," and how do results change under the other judge prompts you tried?
  5. How often does the conflict detector fire on non-contradictory content such as mixed opinions or different time scopes? A small precision/recall annotation of µ_conf on a sample of units would directly address the concern in Weakness 2. Could you also report per-query latency and indexing cost relative to the baselines?
6 comments
AuthorsOpenReview ↗

Response to Reviewer C4m9 (1/5)

We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments have helped us refine the methodological explanations, experimental validation, and interpretation of results. We have carefully examined these concerns, conducted targeted additional experiments, and further clarified the relevant details in the revised manuscript. We respond to the comments below.

Evaluation Reliability and Potential Judge Bias (W1 & Q3)

Thank you for the specific concerns about evaluation reliability and result interpretation. Our original five-dimensional LLM pairwise evaluation uses both presentation orders to assess relative answer quality. To provide a more objective assessment of factual correctness, we have added a gold-reference QA experiment to RQ1 and further analyzed the results.

(1) Gold-reference QA Evaluation. We select 150 test queries each from HotpotQA and 2WikiMultiHopQA and compare FEGR with LightRAG, HippoRAG2, and PathRAG. All methods use the same source corpora, queries, and 6K evidence budget. GPT-4o-mini and Qwen3.5-9B serve separately as full-pipeline LLM backbones, with each method independently performing its native indexing, retrieval, and answer generation. We evaluate answers using normalized Exact Match (EM) and token-level F1 against reference answers.

Table 2. Gold-reference QA performance on two multi-hop QA benchmark subsets (EM/F1, %).

BackboneMethod2WikiMultiHopQA EM2WikiMultiHopQA F1HotpotQA EMHotpotQA F1
Qwen3.5-9BFEGR49.3359.2049.3367.10
Qwen3.5-9BLightRAG40.6749.4238.0053.75
Qwen3.5-9BHippoRAG246.0054.5445.3361.70
Qwen3.5-9BPathRAG22.6727.2541.3357.80
GPT-4o-miniFEGR52.0062.4052.6769.50
GPT-4o-miniLightRAG43.3352.1041.3357.90
GPT-4o-miniHippoRAG249.3359.3048.0065.10
GPT-4o-miniPathRAG25.3331.7543.3360.50

Each benchmark contains 150 test queries per method and backbone. Boldface indicates the best results within each backbone and metric (ties included).

FEGR achieves the highest EM/F1 point estimates in all four benchmark–backbone settings, showing a consistent favorable trend across datasets and LLM backbones. These results are obtained from independently executed full RAG pipelines. Unlike the original five-dimensional LLM pairwise evaluation, gold-reference QA directly compares answers with external reference answers and does not rely on the original judge's subjective criterion-level assessments. Thus, the additional experiment provides independent and complementary empirical support for FEGR's end-to-end performance across model settings.

(2) Coherence and Halo Effect. Thank you for highlighting differences across evaluation dimensions and potential judge bias. Table 1 indeed shows that FEGR's Coherence win rates against PathRAG and HippoRAG2 are 69.0% and 57.4%, respectively, the highest among the five dimensions in those comparisons. We emphasized Supportiveness and Conflict Avoidance because they are directly aligned with FEGR's evidence-reliability modeling objectives, not because they achieve the highest win rates in every comparison. We have made this result explicit in Section 4.2 of the revised manuscript to represent the dimensional performance accurately. Regarding possible LLM judge bias, the original RQ4 component ablations show differentiated changes across evaluation dimensions, while the new gold-reference QA experiment provides a factual-correctness assessment independent of the original five-dimensional judge: FEGR obtains the highest EM and F1 point estimates on both benchmarks under both LLM backbones. Together, these complementary results show that FEGR's favorable performance is not limited to LLM pairwise judgments but is also supported by evaluation against external reference answers.

AuthorsOpenReview ↗

Response to Reviewer C4m9 (2/5)

Reliability Gate and Case Study Consistency (W2 & Q1)

Thank you for carefully examining the consistency between Eq. (4) and the path admission results in the case study. The original Eq. (4) primarily presented the multiplicative aggregation structure of the four evidence conditions. To clarify the complete computation and its parameter settings, we have explicitly incorporated four fixed exponents into the revised equation and provided their values in the appendix, allowing the reliability calculation to be directly reconciled with the path selection results in the case study.

(1) Reliability Calculation. The revised Eq. (4) is:

M_{\mathrm{rel}}(p,q)^{\alpha_{\mathrm{rel}}} M_{\mathrm{sup}}(p,q)^{\alpha_{\mathrm{sup}}} \bigl(1-M_{\mathrm{conf}}(p,q)\bigr)^{\alpha_{\mathrm{conf}}} \bigl(1-M_{\mathrm{unc}}(p,q)\bigr)^{\alpha_{\mathrm{unc}}}. \tag{4}$$ The four fixed exponents are: $$(\alpha_{\mathrm{rel}},\alpha_{\mathrm{sup}}, \alpha_{\mathrm{conf}},\alpha_{\mathrm{unc}}) =(1.00,\;1.00,\;0.65,\;0.50).$$ For the singleton path $p_1=\{u_1\}$ in the case study shown in Appendix Figure 5 and Table 8, the evidence unit has four membership values of $(0.383,0.794,0.650,0.249)$. Substituting these values into the revised equation gives: $$R(p_1,q)=0.383^{1.00}\times0.794^{1.00}\times(1-0.650)^{0.65}\times(1-0.249)^{0.50}\approx0.13319>0.10=\tau^*.$$ Therefore, the path reliability of $p_1$ exceeds the Reliability Gate threshold and satisfies the admission condition, consistent with the path selection result reported in Table 8. **(2) Reliability Gate and Coverage-first Admission.** FEGR first constructs an admissible candidate path pool through the Reliability Gate and subsequently performs the Coverage-first and Reliability-fill selection stages within this pool. Both stages are subject to the same reliability admission condition; Coverage-first does not bypass the Reliability Gate. In this case, $p_1$ first satisfies the reliability threshold and is then prioritized for admission during Coverage-first based on its contribution to covering the query's information requirements.
AuthorsOpenReview ↗

Response to Reviewer C4m9 (3/5)

Membership Scoring and Evaluation Transparency (W3 & Q4)

Thank you for your specific suggestions concerning the reproducibility of FEGR's scoring mechanism and the transparency of its evaluation protocol. We clarify these points based on the original method definitions and the additional details provided in the revised manuscript.

(1) Membership Signals and Parameter Selection. FEGR's four memberships are constructed from predefined operational evidence signals and fixed rules. Relevance, Supportiveness, and Uncertainty use normalized weighted aggregation. Conflict is evaluated over locally comparable evidence pairs sharing the relevant Subject, Attribute, and Scope; conflict responses are computed with fixed rule weights, and the maximum response determines the corresponding risk membership. To improve reproducibility, we have further disclosed the following fixed weights in Appendix A.1 of the revised manuscript.

Table 4. Fixed operational signal and conflict-rule weights.

MembershipFixed Signal / Rule Weights
μrel\mu_{\mathrm{rel}}Retrieval confidence (0.45), typed-anchor coverage (0.35), structural correspondence (0.20)
μsup\mu_{\mathrm{sup}}Core-anchor coverage (0.25), predicate/relation coverage (0.20), requirement coverage (0.20), answer-bearing quality (0.20), assertion density (0.15)
μconf\mu_{\mathrm{conf}}Numeric contradiction (0.90), date contradiction (0.90), entity–value contradiction (0.85), explicit correction (0.80), local negation mismatch (0.65), mutually exclusive status (0.55)
μunc\mu_{\mathrm{unc}}Hedging (0.20), ambiguity (0.15), weak grounding (0.20), low retrieval confidence (0.15), content-form risk (0.15), incomplete scope (0.10), conflict unobservability (0.05)

For Conflict, the weighted rule response is dk(u,v)=λkck(u,v)d_k(u,v)=\lambda_k c_k(u,v). Unlike the other three memberships, λk\lambda_k denotes a conflict-rule importance coefficient and does not participate in normalized weighted averaging. All relevant parameters were fixed before the formal evaluation; the 200 formal evaluation queries were not used to tune membership weights or conflict-rule parameters.

(2) Judge Protocol and Prompt Variants. Regarding the J2 Calibration Variant, we clarify that J2 is an internal version label from the development of the evaluation protocol, not a judge specifically optimized for FEGR. The evaluation protocol was not selected based on FEGR's win rates over baselines, and the label J2 does not imply method-specific evaluation criteria. The formal evaluation applies the same fixed judge prompt and five evaluation criteria to FEGR and all baselines. Each comparison uses anonymous A/B labels, evaluates both A/B and B/A presentation orders, and aggregates judgments in a tie-aware manner to reduce presentation-position bias. We have also standardized the naming in the project materials and revised appendix to maintain consistency among the prompt, evaluation procedure, and reported results.

Comments on the paper

1 comment
Area_Chair_4szVOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewers,

The Author–Reviewer Discussion period is now nearing its end and will close on 18 November. The authors have submitted their response and, in some cases, a revised manuscript.

Could you please take a moment to:

  • Read the authors’ rebuttal and any updated version of the paper;
  • Leave a reply to the authors on the points you raised, indicating whether your concerns have been addressed.

Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.

If you are unable to participate further, please let me know as soon as possible.

Thank you for your time and careful reviewing.

Best regards,

Area Chair, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.