- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how perturbation-based attribution methods (LIME/SHAP-style) assign importance to retrieved paths in knowledge-graph RAG when several paths express the same fact. Through controlled experiments that add alternative expressions of one fact, the authors show that the importance is split across these paths and that removing the top-ranked path alone rarely changes the answer. They propose treating such paths as one group when computing attributions, and they recommend reporting the grouping rule together with the path scores.
Critical strengths
- The paper asks a well-motivated question: how path-level attribution should be read when several retrieved paths carry the same information. This is relevant for anyone who uses such scores to explain KG-RAG answers.
- The experiments are carefully controlled. Only the number of equivalent paths is varied, while the question, reader and other evidence stay fixed. A control condition that adds irrelevant facts about the same entity helps rule out simple context dilution.
- The authors are careful about what their results do and do not show, and the discussion of limitations is candid.
Critical weaknesses
- The practical implications could be made clearer. The main finding is that equivalent paths end up sharing importance, which is largely what one would expect; the analysis in Section 3.1 also predicts it. I was left wondering how this insight could be used in practice, for example to improve answer accuracy, reduce cost, or decide which evidence to keep. Showing even one such downstream benefit would make the contribution much more convincing.
- The paper cites eight references, and its discussion of both attribution methods and graph-based RAG is quite brief. Several relevant lines of work exist: how attribution behaves when inputs are redundant or correlated, how to compute importance for groups of inputs rather than individual ones, and how to attribute RAG outputs to retrieved context by ablating parts of it and fitting a surrogate model. Positioning the paper against these lines of work would make its contribution easier to assess.
- The novelty of alias canonicalization is unclear. Treating equivalent pieces of knowledge as a single unit is a common idea. For example, synonymous entities are often linked or merged when knowledge graphs are built for RAG, and similar entries are consolidated in agent memory. A discussion of how the proposed grouping differs from these practices would help clarify what is new.
- The scope and scale of the experiments are limited. The setting focuses on cases where several paths supporting the same answer are retrieved together, and the equivalent paths in the controlled study are constructed by the authors. The experiments also use a fairly small number of questions per setting and mostly small readers, so it is hard to judge how well the findings generalize.
- I found the paper as a whole difficult to follow. Starting from the abstract and introduction, the writing moves quickly between many specific points without a clear, connected storyline. The overall structure also differs from a typical CS paper; for example, Section 3 combines the method with experimental details. In addition, there is no concrete example of the actual data or the reader prompt. A clearer narrative, a worked example and a cleaner separation between method and experiments would help readers a lot.
Questions for authors
- Could the authors comment on how practitioners might use these findings in a KG-RAG system, for example for evidence selection, efficiency or citation? A small example would be very helpful.
- How does alias canonicalization relate to existing ways of merging equivalent knowledge, such as entity deduplication in graph-based RAG or group-level attribution?
- Could the authors share the reader prompt, the format of the path text, and one real example from the data?
- How many questions were filtered out by the selection criteria, and do the authors expect the findings to hold on larger question sets?