- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether GraphRAG increases relational metadata leakage compared with conventional text-based RAG. It introduces relational canaries and matched controls, separates retrieval exposure from conditional reproduction, and proposes ProvPatch, a provenance-aware method that regenerates index artifacts affected by sensitive-data removal. Across three multi-hop QA datasets, GraphRAG increases average exact extraction from 46.4% to 75.1% compared with dense RAG. The analysis suggests that this increase primarily results from more frequent retrieval of protected identifiers. ProvPatch reduces observed exact extraction to zero in the five-turn canary audit while regenerating 9.8–14.1% of indexed artifacts. However, the natural-span experiments show higher residual extraction and larger utility losses.
Critical strengths
- The analysis helps explain the mechanism behind increased leakage. The paper matches the corpus, generator, prompts, and evidence budget across retrieval conditions, then measures retrieval exposure and conditional reproduction separately.
- The repair framework addresses a practical problem involving derived artifacts. The paper demonstrates that sensitive identifiers can persist in relation descriptions, community reports, and other artifacts after their source occurrence is removed. Comparing local regeneration with source-only updates and full rebuilding, while reporting extraction, utility, and update cost, provides a useful operational evaluation.
Critical weaknesses
- The repair results have limited validation beyond controlled synthetic identifiers. ProvPatch relies on a recorded dependency graph, but the treatment of entity merging, community changes, and newly affected dependencies is underspecified. The reported regeneration fraction is difficult to interpret without knowing the number and distribution of targets patched together.
- The comparison with existing attacks and the characterization of adaptivity need clarification. Section 2.2 briefly summarizes prior extraction attacks without sufficiently distinguishing their attacker knowledge, objectives, and interaction strategies. This makes it difficult to assess the methodological novelty of the proposed audit. Additionally, Table 5 specifies a fixed five-turn prompt sequence that receives conversation history, rather than an attacker that selects subsequent queries based on observed responses. Consequently, the reported results establish effectiveness against this particular multi-turn protocol, while providing more limited evidence against adaptive extraction strategies.
Questions for authors
- How stable are the main results across independent index builds and target samples? Please provide variability estimates or confidence intervals for leakage amplification, post-patch recovery, and regeneration costs, accounting for multiple prompts targeting the same record. This would help establish whether the findings are robust to variation in graph construction and target selection.
- How many targets are patched in each efficiency experiment, and are these all injected targets or only successfully extracted targets?
- How much of the leakage gap remains if dense retrieval is given access to the same derived relation descriptions and community reports as GraphRAG, without graph traversal? The current comparison changes both the indexed representations and the retrieval policy. This control would help distinguish amplification caused by relational materialization from amplification caused by graph-based retrieval.
2 comments
Thank you for the detailed response and additional experiments.
The statistical analysis remains conditional on a single index build, and the robustness of ProvPatch under structural graph changes is still not fully evaluated.
Overall, the rebuttal addresses most of my concerns, but some limitations remain. I will therefore keep my high score.
Thank you for recognizing the retrieval-stage analysis and the practical value of repairing derived artifacts.
W1 & Q2: Provenance propagation and patch workload.
ProvPatch patches targets whose disclosure is confirmed by the audit (Section 4.4). It merges their recorded source-to-artifact dependency closures, edits the source occurrences, and regenerates affected artifacts in dependency order. Each shared descendant is regenerated at most once (Eq. 14; Algorithm 1). The evaluated repair follows dependencies recorded during index construction; Section 6 discusses closure expansion for global changes involving entity merging and community repartitioning. Across the three datasets, ProvPatch regenerates 9.8-14.1% of artifacts using 11.2-15.1% of full-rebuild time (manuscript Table 20). We will release the audit code, patch-target manifests, and representative repair traces.
W2: Attack setting and multi-turn protocol.
Liu et al.[1] study raw-text and structured-information extraction; AGEA[2] adaptively reconstructs graphs under a query budget; and GRASP[3] reconstructs subgraphs under prompt-based safeguards. Our auditor knows the entity endpoints and relation type, while the protected identifier remains hidden. The audit combines traceable relational targets, matched retrieval comparisons, retrieval-generation decomposition, and provenance-guided repair. Its interaction protocol is a fixed five-turn audit with dialogue history, applied consistently across retrieval systems and before and after patching.
Q1: Statistical uncertainty.
In the single-turn audit, the absolute Graph-Dense exact-extraction rate differences are 31.0%, 28.4%, and 26.8% on HotpotQA, 2WikiMHQA, and MuSiQue, respectively. After ProvPatch, the five-turn audit records 0/4,500 exact extractions and 9/4,500 normalized recoveries. Table R3 reports dataset-specific, two-sided 95% Clopper-Pearson intervals, treating each complete five-turn target episode as one observation. The exact-recovery interval is [0, 0.246]% in each dataset. These intervals quantify episode-level uncertainty conditional on the evaluated index.
Table R3: Post-patch recovery and two-sided 95% confidence intervals. Counts follow manuscript Table 13. Each complete five-turn target episode is one observation. The binomial intervals assume independent target episodes and are conditional on the evaluated index.
Q3: Dense retrieval over shared derived artifacts.
DenseShared indexes the same source text units, entity/relation descriptions, and community reports available to GraphRAG, with the embedding model, generator, prompts, and evidence budget held fixed. Mean dense-retrieval ESR increases from 46.4% to 63.7%, reducing the absolute gap to GraphRAG from 28.7% to 11.4% (Table R4).
Table R4: Shared-artifact control for single-turn exact extraction (ESR, %). Remaining gap is GraphRAG ESR minus DenseShared ESR.
[1] Liu J, Zhang J, Wang S. Exposing privacy risks in graph retrieval-augmented generation[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026: 18073-18093.
[2] Yang S, Zhang J, Wang Y, et al. Query-efficient agentic graph extraction attacks on graphrag systems[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 16017-16041.
[3] Song M, Kim J, Kang M, et al. Graphs Don't Stay Secret: Practical Subgraph Reconstruction Attacks on Defended Graph RAG[J]. arXiv preprint arXiv:2602.06495, 2026.