acceptodds
Under review as a conference paper at ICLR 2027

Auditing Source Provenance in Retrieval-Augmented Data Synthesis

Abstract

Retrieval-augmented data synthesis transforms source documents into new datasets, making their provenance difficult to verify. We study whether document owners can audit such downstream use by modifying only their source documents, without controlling retrieval or generation or editing the synthesized outputs. We introduce a source-side semantic watermarking framework in which a secret key selects recommendations among alternatives with equivalent behavior for a given task. The key is encoded in recommendation relations, allowing the signal to be expressed without verbatim reproduction of source passages. Given access to a candidate derived dataset, a relation-aware auditor extracts expressed recommendations where available and aggregates group-level evidence to test alignment with the owner's key. We evaluate the framework under blind synthesis, where the generator is not instructed to preserve or reveal the watermark. Matched independent-key controls distinguish key-specific transfer from incidental generator preferences. Experiments on controlled technical-document benchmarks, including executable tasks and held-out content groups, demonstrate detectable source-specific signals after synthesis. Further analyses separate relation coverage from fidelity and characterize detection under partial source use, rewriting, and data mixing, as well as removal through informed filtering. These findings establish the feasibility of source-side provenance auditing for RAG-derived datasets and characterize the conditions under which its evidence is preserved or lost.

Primary area:
datasets and benchmarks
Keywords:
Data Provenance, Semantic Watermarking, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_82HaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RelMark. The method embeds private patterns through recommendations between task-equivalent alternatives, and later audits a completed QA dataset by recovering these recommendation directions and testing their alignment with a private key. The paper evaluates the method on CLI, API, and SQL documentation across four generators, and studies rewriting, shuffling, content selection, and targeted editing.

Critical strengths

The paper studies an important problem: auditing source use after retrieval-augmented synthesis when generation logs and live querying are unavailable. The key-blind recovery protocol and independent-key controls are also carefully designed, and the experiments are more thorough than a simple proof-of-concept.

Critical weaknesses
  1. RelMark does not robustly establish source provenance. Its signal drops to 0/80 votes under functional-facts synthesis, while useful factual QA is preserved; local editing likewise removes the signal while retaining operational facts. This suggests RelMark traces a particular recommendation rather than actual source use.

  2. The setting is highly specialized and intrusive. The median carrier occupies 31.85% of the keyed source, and the method requires verifiably interchangeable alternatives, which largely confines the evaluation to CLI/API/SQL documentation.

  3. The baseline comparison is insufficient. Comparing mainly against KGW (Kirchenbauer et al., 2023) does not establish an advance over semantic or source-attribution methods that are explicitly designed to survive rewriting, such as SemStamp (Hou et al., 2024), WASA (Lu et al., 2025), and SemTrace (Zhang et al., 2026). The current result largely shows that semantic content survives semantic transformation better than token bias.

[1] Hou et al., SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation, NAACL 2024. [2] Lu et al., WASA: WAtermark-based Source Attribution for Large Language Model-Generated Data, ACL Findings 2025. [3] Zhang et al., SemTrace: Source-Grounded Semantic Signatures for Tracing LLM Exposure to Protected Documents, 2026.

Questions for authors
  1. How should RelMark be interpreted as a source-provenance method when the paper itself shows that a benign change in synthesis objective can eliminate the signal while preserving useful source-derived facts?
  2. Can the authors demonstrate that RelMark remains effective when the carrier occupies a much smaller fraction of the source, and on domains where executable interchangeable alternatives are not naturally available?
  3. Why is there no direct empirical comparison with semantic watermarking or semantic source-attribution methods, rather than primarily with KGW?
Reviewer_ZT3WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RelMark, which embeds source markers through recommendation preferences between task-equivalent alternatives and audits their presence in completed synthetic QA datasets. Experiments show that the method can recover provenance signals from technical documentation and preserve them through some rewriting.

Critical strengths

The work addresses a practical provenance problem in synthetic data, uses recommendation preferences to carry signals that can survive rewriting, and validates source specificity through reasonably thorough control experiments.

Critical weaknesses
  1. The paper shows noticeable signs of AI generation in its presentation. Figure 1 contains visual artifacts, including uneven colors and fills. The text also contains repetitive statements and dense terminology, making the core method and main findings less prominent.

  2. The main experiments involve a relatively small collection of independent source groups. Generalization to more software projects and larger source collections remains unclear.

  3. The mixed-data experiments mainly resample existing QA pools and do not sufficiently evaluate actual multi-source retrieval and data synthesis pipelines.

  4. The reported marker-to-source text ratio in the native-reference setting is approximately 31.85%. The observed effectiveness may depend on this relatively high marker density.

Questions for authors
  1. Figure 1 shows noticeable AI-generated visual artifacts, including uneven colors and fills. Could it be manually edited or redrawn using PowerPoint or draw.io to improve its visual quality?

  2. Does detection performance remain stable across more software projects and substantially larger collections of independent source groups?

  3. How well does the method perform when a small proportion of marked documents is mixed into a large document corpus and retrieval and QA synthesis are performed again?

  4. Does the method remain effective with longer documents and lower marker density?

Reviewer_QZGuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of establishing source provenance for secondary datasets generated from original datasets, such as QA pairs from documentation. In such cases, the synthesis generation pipeline may not retain provenance information injected by the original dataset authors. The paper proposes RelMark, which makes use of functionally equivalent operations to encode private keys that can subsequently be checked against to establish provenance via aggregation of QAs linked to specific groups and statistical tests. Experiments were done and compared against KGW watermarking.

Critical strengths
  • The paper identifies a niche opportunity to embed watermarking signal, where there are verified functionally equivalent choices to specific content contained in documents. This allows embedding of private key that serve as provenance signals, into the technical content itself, making it resistant to attacks such as paraphrasing.
Critical weaknesses
  • The problem setting also seem narrow and applicability of the method appears small in practice. RelMark seems to only work for settings such as technical documentation where there can be established functionally equivalent options. Even for the settings that they highlight such as programming frameworks, different commands could have nuanced but important differences in their operations unless heavily scoped down to limited qualifications, resulting in fragility of the method's applicability.
  • The method is vulnerable to routine synthesis pipeline variations as the paper pointed out. E.g. just getting the model to generate purely objective facts destroys the signal completely. Such prompting for RAG data synthesis is standard practice. Similarly, targeted QA relation editing has high negative impact.
  • The method omits important watermarking baseline and only tested on basic KGW which is not well suited for this problem. The authors should discuss and test against data-centric provenance watermarking and those designed to be robust against code and paraphrasing such as [1-3].
  • The paper is very hard to understand with its very dense writing with repetitive jargon and little narrative and intuitive explanations, likely due to over-reliance of AI assisted writing.

[1] Lau et al, Waterfall: Scalable Framework for Robust Text Watermarking and Provenance for LLMs, EMNLP 2024 [2] Lee et al, Who Wrote this Code? Watermarking for Code Generation, ACL 2024 [3] Hou et al, SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation, NAACL 2024

Questions for authors
  • How does RelMark compare to other key relevant baselines such as [1-3] as explained in the section above?
  • How can the method be robust to variations in the pipelines used for synthesis, especially when the paper has shown vulnerability to standard prompts such as basic instructions to output objective facts or other targeted editing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.