acceptodds
Under review as a conference paper at ICLR 2027

Particle-KGRAG: Answer-Aligned Confidence for Knowledge-Graph Question Answering

Abstract

Knowledge-graph question answering requires confidence scores that distinguish correct answers from retrieval failures, yet a score used to guide graph search need not predict answer correctness. We introduce Particle-KGRAG, a training-free construction of answer-aligned confidence. It samples independent paths from a Personalized-PageRank-guided graph prior, scores completed paths with a terminal LLM judge, and aggregates normalized weights by extracted answer. Confidence is the posterior mass assigned to the answer actually returned. Using the path prior as the proposal gives a self-normalized importance-sampling formulation with a fixed budget of 129 LLM calls per query. Across six benchmarks and three seeds, with post-registration analysis changes disclosed, mean correctness AUROC is 70.6 versus 63.7 for a ToG-style beam's search-derived score; improvements on HotpotQA, 2WikiMultiHopQA, and CWQ survive paired and query-clustered testing with false-discovery-rate correction. A separately pre-registered rerun correcting prompt rendering gives 66.4 versus 58.7 and preserves these three improvements. On archived 2WikiMultiHopQA runs, selective accuracy at 50% coverage increases from .337 to .573, while the full-coverage accuracy difference is not statistically established. A beam control with answer-level aggregation reduces the AUROC gap on that dataset to 1.3 points, leaving the contribution of independent sampling unresolved. There is no demonstrated answer-accuracy gain or corrected superiority over a perturbation-based uncertainty baseline, and MetaQA-3hop favors the beam. These results characterize the utility and limits of answer-aligned confidence from judge-weighted KG paths.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
knowledge-graph question answering, uncertainty quantification, selective prediction, confidence estimation, self-normalized importance sampling, conformal prediction, retrieval-augmented generation, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n327OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free confidence estimation method for KGQA by sampling PPR-guided paths, assigning terminal LLM-judge weights, and aggregating normalized weights by answer.

Critical strengths

The paper explicitly distinguishes path quality from answer correctness.

The answer-level aggregation mechanism is simple and does not require task-specific training.

Critical weaknesses
  1. Posterior mass is not necessarily a measure of answer correctness.

The method aggregates judge-weighted paths into an answer-level confidence score, but the judge does not observe the extracted candidate answer.

  1. Answer selection is inconsistent with answer aggregation.

The method returns the answer associated with the highest-weight individual particle rather than the answer with the largest aggregated mass.

Questions for authors

What assumptions justify interpreting judge-weighted path mass as answer correctness when the judge never evaluates the extracted candidate answer?

Reviewer_GZDKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper targets knowledge-graph question answering (KGQA) and presents Particle-KGRAG, an approach for deriving an answer-aligned confidence value. The method starts from a seed question entity, establishes a path prior via personalized PageRank (PPR) and self-normalized importance sampling (SNIS), and samples candidate answer paths. An LLM judge then provides a score for a given path, which is aggregated per answer. Experiments are conducted on six different benchmarks, primarily comparing against ToG (an LLM-guided beam search approach) as well as state-of-the-art methods R^2C and R^2-KG. The results report positive trends in confidence quality compared to ToG on four of the six datasets, while answering performance generally degrades. Furthermore, selective accuracy is lower than that of R^2C and R^2-KG.

Critical strengths
  • The full experimental protocol was frozen upfront, and the paper transparently reports the negative findings of these experiments.
Critical weaknesses
  • The impact of the paper is limited. The proposed methodology does not consistently outperform the beam-search baseline (ToG) in confidence quality, and shows a clear performance gap to state-of-the-art methods in answer quality. Comparison against the State of the Art (R^2C, R^2KG) in Figure 2 shows a performance gap of the proposed method in selective accuracy. Most importantly, the proposed approach does not appear to be applicable to state-of-the-art methods following the retrieval-augmented generation (RAG) paradigm.

  • The methodological novelty is limited as the paper combines several established components: personalized PageRank, an LLM judge, and self-normalized importance sampling (SNIS).

Questions for authors
  • How does the LLM call budget affect the results? Ablation studies with different LLM budgets would be useful.

  • Could the approach be extended to more realistic setups in which the seed entity is unknown?

Minor comments:

  • ToG is not cited when first mentioned.
Reviewer_UzpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Particle-KGRAG builds a training-free confidence score for KGQA. It samples 32 PPR-guided paths, weights each path with an LLM judge, and sums the normalized weights per extracted answer. Confidence is the mass on the returned answer, which the paper frames as self-normalized importance sampling. Across six benchmarks, mean AUROC is 70.6 vs. 63.7 for a ToG-style beam's search score, with significant gains on HotpotQA, 2Wiki and CWQ. On 2Wiki, selective accuracy at 50% coverage rises from .337 to .573.

Critical strengths
  1. Exemplary rigor and transparency: a pre-registered protocol with all deviations listed, paired and query-clustered tests with FDR correction, against-interest reporting, and a shipped checker for the headline numbers.
  2. A useful finding. Search-derived beam scores can rank correctness below chance (2Wiki AUROC 46.4), and answer-level confidence substantially improves correctness ranking and selective prediction.
Critical weaknesses
  1. The contribution of the sampling method proposed in this study remains unclear. On the "2Wiki" dataset, employing a beam answer-level aggregation strategy reduces the gap from 30.9 points to 1.3 points; conversely, when the answer is held constant, no performance difference is observed across any of the datasets (Section 5.3). Consequently, the performance gains appear to stem primarily from "answer-level aggregation" rather than from the PPR/SNIS sampling itself.

  2. Furthermore, the proposed method yields lower accuracy on 5 out of the 6 datasets. Additionally, since the AUROC for each system is calculated based on its own generated answers, the results are not directly comparable. The paper also fails to include simple baseline methods—such as "self-consistency" or "semantic entropy"—that would entail a comparable level of computational resource consumption.

Questions for authors
  1. At matched depth and judge prompts, what does independent PPR sampling add over answer aggregation on beam paths?
  2. How does a K=32 self-consistency or semantic-entropy baseline compare at the same 129-call budget, ideally with an accuracy-matched analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.