acceptodds
Under review as a conference paper at ICLR 2027

One Record, Many Artifacts: Relational Metadata Leakage in GraphRAG

Abstract

GraphRAG transforms document collections into retrievable entities, relations, and summaries, enabling effective reasoning over relational evidence but also creating a new privacy surface: metadata that is weakly salient in source text can become directly addressable once materialized into graph-derived index artifacts. To study and mitigate this risk, we present a unified audit-and-repair framework for relational metadata leakage in GraphRAG. The framework combines relational canaries and matched controls for precisely traceable leakage auditing, retrieval-generation decomposition and representation analysis for identifying the mechanism of amplification, and ProvPatch, a provenance-aware repair method that traces confirmed targets through their source-to-index dependency closure and regenerates only affected artifacts. Across three multi-hop QA benchmarks, GraphRAG increases exact extraction by 28.7 percentage points over matched dense RAG. Our analyses show that this amplification arises primarily from relation-level materialization and increased retrieval exposure, rather than from a substantial change in conditional reproduction once the target has been retrieved. Under a five-turn adaptive attack, ProvPatch reduces exact extraction to zero and normalized identifier recovery to 0.002 while recomputing only 9.8-14.1% of the index. These results characterize relational materialization as a distinct source of metadata exposure in GraphRAG and show that provenance-guided local regeneration can repair this exposure without rebuilding the full index.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Graph Retrieval-augmented Generation, Privacy Leakage, Provenance-Aware Repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mgs4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether GraphRAG increases relational metadata leakage compared with conventional text-based RAG. It introduces relational canaries and matched controls, separates retrieval exposure from conditional reproduction, and proposes ProvPatch, a provenance-aware method that regenerates index artifacts affected by sensitive-data removal. Across three multi-hop QA datasets, GraphRAG increases average exact extraction from 46.4% to 75.1% compared with dense RAG. The analysis suggests that this increase primarily results from more frequent retrieval of protected identifiers. ProvPatch reduces observed exact extraction to zero in the five-turn canary audit while regenerating 9.8–14.1% of indexed artifacts. However, the natural-span experiments show higher residual extraction and larger utility losses.

Critical strengths
  1. The analysis helps explain the mechanism behind increased leakage. The paper matches the corpus, generator, prompts, and evidence budget across retrieval conditions, then measures retrieval exposure and conditional reproduction separately.
  2. The repair framework addresses a practical problem involving derived artifacts. The paper demonstrates that sensitive identifiers can persist in relation descriptions, community reports, and other artifacts after their source occurrence is removed. Comparing local regeneration with source-only updates and full rebuilding, while reporting extraction, utility, and update cost, provides a useful operational evaluation.
Critical weaknesses
  1. The repair results have limited validation beyond controlled synthetic identifiers. ProvPatch relies on a recorded dependency graph, but the treatment of entity merging, community changes, and newly affected dependencies is underspecified. The reported regeneration fraction is difficult to interpret without knowing the number and distribution of targets patched together.
  2. The comparison with existing attacks and the characterization of adaptivity need clarification. Section 2.2 briefly summarizes prior extraction attacks without sufficiently distinguishing their attacker knowledge, objectives, and interaction strategies. This makes it difficult to assess the methodological novelty of the proposed audit. Additionally, Table 5 specifies a fixed five-turn prompt sequence that receives conversation history, rather than an attacker that selects subsequent queries based on observed responses. Consequently, the reported results establish effectiveness against this particular multi-turn protocol, while providing more limited evidence against adaptive extraction strategies.
Questions for authors
  1. How stable are the main results across independent index builds and target samples? Please provide variability estimates or confidence intervals for leakage amplification, post-patch recovery, and regeneration costs, accounting for multiple prompts targeting the same record. This would help establish whether the findings are robust to variation in graph construction and target selection.
  2. How many targets are patched in each efficiency experiment, and are these all injected targets or only successfully extracted targets?
  3. How much of the leakage gap remains if dense retrieval is given access to the same derived relation descriptions and community reports as GraphRAG, without graph traversal? The current comparison changes both the indexed representations and the retrieval policy. This control would help distinguish amplification caused by relational materialization from amplification caused by graph-based retrieval.
2 comments

Thank you for recognizing the retrieval-stage analysis and the practical value of repairing derived artifacts.

W1 & Q2: Provenance propagation and patch workload.

ProvPatch patches targets whose disclosure is confirmed by the audit (Section 4.4). It merges their recorded source-to-artifact dependency closures, edits the source occurrences, and regenerates affected artifacts in dependency order. Each shared descendant is regenerated at most once (Eq. 14; Algorithm 1). The evaluated repair follows dependencies recorded during index construction; Section 6 discusses closure expansion for global changes involving entity merging and community repartitioning. Across the three datasets, ProvPatch regenerates 9.8-14.1% of artifacts using 11.2-15.1% of full-rebuild time (manuscript Table 20). We will release the audit code, patch-target manifests, and representative repair traces.

W2: Attack setting and multi-turn protocol.

Liu et al.[1] study raw-text and structured-information extraction; AGEA[2] adaptively reconstructs graphs under a query budget; and GRASP[3] reconstructs subgraphs under prompt-based safeguards. Our auditor knows the entity endpoints and relation type, while the protected identifier remains hidden. The audit combines traceable relational targets, matched retrieval comparisons, retrieval-generation decomposition, and provenance-guided repair. Its interaction protocol is a fixed five-turn audit with dialogue history, applied consistently across retrieval systems and before and after patching.

Q1: Statistical uncertainty.

In the single-turn audit, the absolute Graph-Dense exact-extraction rate differences are 31.0%, 28.4%, and 26.8% on HotpotQA, 2WikiMHQA, and MuSiQue, respectively. After ProvPatch, the five-turn audit records 0/4,500 exact extractions and 9/4,500 normalized recoveries. Table R3 reports dataset-specific, two-sided 95% Clopper-Pearson intervals, treating each complete five-turn target episode as one observation. The exact-recovery interval is [0, 0.246]% in each dataset. These intervals quantify episode-level uncertainty conditional on the evaluated index.

Table R3: Post-patch recovery and two-sided 95% confidence intervals. Counts follow manuscript Table 13. Each complete five-turn target episode is one observation. The binomial intervals assume independent target episodes and are conditional on the evaluated index.

DatasetExact recoveryNormalized recoveryNIR 95% CI (%)
HotpotQA0/15002/1500[0.016, 0.481]
2WikiMHQA0/15004/1500[0.073, 0.681]
MuSiQue0/15003/1500[0.041, 0.583]

Q3: Dense retrieval over shared derived artifacts.

DenseShared indexes the same source text units, entity/relation descriptions, and community reports available to GraphRAG, with the embedding model, generator, prompts, and evidence budget held fixed. Mean dense-retrieval ESR increases from 46.4% to 63.7%, reducing the absolute gap to GraphRAG from 28.7% to 11.4% (Table R4).

Table R4: Shared-artifact control for single-turn exact extraction (ESR, %). Remaining gap is GraphRAG ESR minus DenseShared ESR.

DatasetDenseDenseSharedGraphRAGRemaining gap
HotpotQA50.869.781.812.1
2WikiMHQA46.363.874.710.9
MuSiQue42.057.668.811.2
Mean46.463.775.111.4

[1] Liu J, Zhang J, Wang S. Exposing privacy risks in graph retrieval-augmented generation[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026: 18073-18093.

[2] Yang S, Zhang J, Wang Y, et al. Query-efficient agentic graph extraction attacks on graphrag systems[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 16017-16041.

[3] Song M, Kim J, Kang M, et al. Graphs Don't Stay Secret: Practical Subgraph Reconstruction Attacks on Defended Graph RAG[J]. arXiv preprint arXiv:2602.06495, 2026.

Thank you for the detailed response and additional experiments.

The statistical analysis remains conditional on a single index build, and the robustness of ProvPatch under structural graph changes is still not fully evaluated.

Overall, the rebuttal addresses most of my concerns, but some limitations remain. I will therefore keep my high score.

Reviewer_5zQoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the leakage of relational metadata in GraphRAG. The authors propose an auditing and repair framework. They insert self-generated relational identifiers into three datasets and compare the information extraction rates of different retrieval systems under matched settings. The results show that GraphRAG has a higher extraction rate than dense RAG. The authors further analyze the retrieval and generation stages separately. They try to show that this difference mainly comes from the fact that target information is more likely to enter the retrieved context.

The authors also propose ProvPatch, which uses the dependency relationships between source data and index artifacts. After the source content is modified, ProvPatch can locate, invalidate, and regenerate the affected derived artifacts. The experiments evaluate the exact extraction rate, normalized identifier recovery rate, downstream question-answering performance, and update cost. The authors also conduct additional tests with natural text snippets, different frameworks, and different generation models.

Critical strengths
  1. The authors separately measure the probability that the target information enters the retrieved context and the probability that the model outputs the target information after it is retrieved. The results support that the observed leakage difference mainly comes from increased retrieval exposure. This makes the analysis not only focus on the comparison of final extraction rates. The use of multiple metrics also improves the credibility of the paper.

  2. The experiments are relatively comprehensive. The paper covers three datasets, two graph retrieval frameworks, and two generation models. It also evaluates information recovery, downstream question-answering utility, and update cost.

Critical weaknesses
  1. The authors argue that the leakage amplification in GraphRAG mainly comes from the materialization of relational information. They support this explanation by comparing ordinary identifiers, natural-language cross-references, and structured relation records. However, these settings change several factors at the same time, including entity and relation information, field formats, and the degree of matching with queries. They do not only change the factor of relation materialization. Therefore, this experiment is not enough to prove that relation materialization is the main cause of the observed difference.

  2. The authors compare the question-answering results before and after removing the injected identifiers to show that the repair can preserve most of the system utility. However, these random identifiers are artificially added to the data. Also, the experiment on removing natural text snippets shows a clear performance drop. Therefore, these results cannot prove that the question-answering performance can still be maintained after removing information from a system where the information is originally meaningful and useful.

  3. The purpose of RAG itself is to retrieve information. If a RAG system returns confidential information marked by a business process, it does not necessarily mean that the RAG system has a security problem. It may also mean that the RAG system performs retrieval well. At least, the authors should explain the difference between the risk measured in this paper and normal information access.

Questions for authors
  1. For the performance drop caused by DNS deletion, how much comes from the removal of the target evidence itself, and how much comes from the side effects on unrelated question-answering tasks?

  2. Why is the NIR of full rebuild not zero?

1 comment

Thank you for the detailed comments on mechanism attribution, utility, and the privacy boundary.

W1: Attribution to relational materialization.

The largest difference observed is at the retrieval stage. Table 2 in the manuscript shows retrieval exposure of 58.2% for Dense RAG and 90.4% for GraphRAG, compared with conditional reproduction of 87.3% and 90.5%. The representation ablation jointly varies relation content, field format, and query alignment. These results support retrieval-side amplification under the combined representation and retrieval configuration.

W2 & Q1: Utility after removing useful information.

The utility cost depends on the information removed. On HotpotQA, canary redaction changes F1 from 71.5% to 70.2%, whereas DNS redaction changes F1 from 68.6% to 53.8% (Table R1). Therefore, the utility-preservation result applies to injected identifier removal. The DNS results quantify the total utility change after editing natural content. Decoupling the effects on different QA questions requires grouping them by overlap with gold supporting evidence and comparing update strategies on the same edited corpus.

Table R1: QA F1 (%) on the original evaluation split, from manuscript Table 16. DNS denotes designated natural sensitive spans.

Index conditionHotpotQA2WikiMHQAMuSiQue
Base68.658.638.5
Canary71.561.444.7
Canary + redaction70.261.743.1
Canary + record-line removal69.159.641.0
DNS + redaction53.846.331.0
DNS + record-line removal48.740.225.2

W3: Disclosure risk and ordinary retrieval.

The threat model permits access to surrounding facts while the operator designates selected internal fields as non-disclosable. The removal setting makes this boundary operational: source/text-unit refresh leaves mean exact extraction at 60.8% (Table R2), while protected identifiers remain in derived artifacts (manuscript Table 19). ProvPatch propagates source edits through recorded dependencies. The audit measures designated-field exposure and the propagation of removal across derived artifacts.

Table R2: Fixed five-turn audit and update cost. Rates and regenerated artifact fractions are percentages; time is relative to full rebuilding. Values are rounded equal-weight means of manuscript Table 13.

Update strategyExact ESRNIRArtifactsTime
Source/text-unit refresh only60.866.901.90.018
Redaction + full rebuild0.00.09100.01.000
Line removal + full rebuild0.00.02100.01.000
ProvPatch0.00.2012.20.132

Q2: Nonzero NIR after full rebuilding.

NIR matches the complete target identifier after reversible normalization, including recoveries missed by exact matching. Across 4,500 five-turn episodes per update strategy, full rebuilding yields zero exact extractions, with four normalized recoveries after redaction and one after record-line removal (manuscript Table 13). These normalized positives count as residual disclosure under the evaluated protocol.

Reviewer_zDxtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a framework to study leakage of relational metadata in GraphRAG. It introduces relational canaries and a method ProvPatch, a repair mechanism.

Critical strengths

The introduction of relational canaries is novel.

Good breakup of leakage metrics (Sec 3.3)

Results are good

Critical weaknesses

The paper does not adequately explain why should there be a significant difference in performance between Dense RAG and GraphRAG if their configurations are the same.

The paper suffers from dense writing. Examples are lacking all throughout the paper. For example, it is hard to understand Sec 4.3.

The problem statement also should have been explained with examples.

Too much important information is included in the Appendix. Thus, it is hard to understand the main paper. Lines 372-373 are prime examples.

Questions for authors

The paper does not adequately explain why should there be a significant difference in performance between Dense RAG and GraphRAG if their configurations are the same.

2 comments

Thank you for recognizing the relational canaries and highlighting the need for clearer examples and presentation.

W1 & Q1: Multi-hop corpora and retrieval exposure.

Our corpora come from HotpotQA, 2WikiMultihopQA, and MuSiQue, multi-hop QA benchmarks that emphasize cross-document relations. These relations make graph-based retrieval a natural design choice; HippoRAG and HippoRAG 2 show improved multi-hop retrieval over dense baselines[1,2]. Matching fixes the corpus, text embedding model, generator, prompts, decoding settings, and evidence budget. The indexed representations and retrieval policies differ: Dense RAG retrieves source chunks, whereas GraphRAG can retrieve entity/relation descriptions and community reports. These additional representations can make target-bearing relations accessible through more retrieval paths. In our audit, HotpotQA retrieval exposure is 58.2% versus 90.4%, while conditional reproduction is 87.3% versus 90.5%. The observed extraction gap is primarily reflected in retrieval exposure.

W2 & W4: Clearer explanations and precise appendix references.

We agree and will provide clearer main-text explanations within the page limit and provide full details in the appendix. In particular, Section 4.3 will explain DNS through three steps: selecting an existing text span, editing its source occurrence, and checking its derived copies (Appendix G; QA utility in Appendix H). We will strengthen the explanations of the experimental setup and canary construction in the revised manuscript, and replace the general reference at lines 372-373 with explicit cross-references to Appendix A.1 and Appendix B.

W3: An end-to-end case study.

We will add an appendix case study and link it from the problem formulation. Using one audited target, it will show the source record, the derived artifacts that retain the target, the retrieved evidence and generated disclosure, and the results of source-only refresh and ProvPatch.

[1] Gutiérrez B J, Shu Y, Gu Y, et al. Hipporag: Neurobiologically inspired long-term memory for large language models[J]. Advances in neural information processing systems, 2024, 37: 59532-59569.

[2] Gutiérrez B J, Shu Y, Qi W, et al. From rag to memory: Non-parametric continual learning for large language models[J]. Proceedings of the 42nd International Conference on Machine Learning, 2025.

Reviewer_zDxtOpenReview ↗

Same scores

Thank you for the responses.

However, at this stage, without actually seeing the example and clearer writing, nothing much can be commented.

I will retain the scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.