acceptodds
Under review as a conference paper at ICLR 2027

Resilient RAG Dataset Inference via Orthogonal Dual-Layer Watermarking

Abstract

Retrieval-Augmented Generation (RAG) allows unauthorized service providers to exploit proprietary text corpora while hiding provenance behind neural paraphrasing and black-box access, making dataset theft difficult to prove. Lexical watermarks wash away under deep rewriting; injected semantic facts survive paraphrasing but are easily pruned by content filters, or are prohibited in regulated domains such as medicine and law where facts cannot be altered. The Thief's Dilemma formalizes a carrier-conditional bottleneck: under joint carrier contraction, suppressing both channels violates any utility requirement above the carrier-free factual rate. Fact-only reconstruction lies outside this family and can erase regulated-mode provenance. We present orthogonal dual-layer watermarking with a black-box Prober-Verifier. The semantic layer embeds mode-specific invariants: open-domain documents receive long-tail predicate bindings, whereas regulated documents receive truth-preserving coordinate permutations that leave the causal claim graph strictly intact. With semantics frozen, Bidirectional Masked Contextual Resampling (BMCR) introduces tokenizer-independent functional-word biases into disjoint syntactic slots. An active prober uses at most 40 target-side API calls, fusing complementary evidence via a directional Gaussian-copula test. Across a 150-setting sweep of carrier-retaining attacks on the RDI benchmark, our joint auditor maintains at least 98.17% detection power at a 0.1% false-positive rate. Under an aligned 40-call budget, dual-layer attribution reaches 99.35% power against adaptive mixed attacks, exceeding the active semantic channel alone by 15.27 percentage points. Finally, we provide finite text-edit certificates, prove noninferiority on downstream clinical and legal tasks, and show that amortized calibration reduces attribution overhead to 110.68 calls per audit in production monitoring. Code is available at https://anonymous.4open.science/r/rpd_anonymous-F278.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Retrieval-Augmented Generation, Intellectual Property Protection, Dataset Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ffYsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RAG dataset inference: deciding from black-box answers whether a suspect RAG service uses a protected corpus. It proposes a dual-layer watermark. The semantic layer injects long-tail fact bindings in open domains, or uses truth-preserving reorderings in regulated domains. The lexical layer, BMCR, biases functional words in a tokenizer-independent way. A 40-call Prober-Verifier fuses the two channels with a Gaussian-copula test at α = 10⁻³. The theory gives a "Thief's Dilemma" bound under an assumed joint carrier contraction, plus query-budget certificates. Experiments are mainly on RDI, a new LLM-generated benchmark, with FARAD and Wikinews as external tests. The method reaches 97.1% [email protected]% FPR on RDI-Hard, at least 98.17% detection across a 150-setting attack sweep, and a 15.27-point gain over semantic-only on a mixed-attack workload. The paper also reports calibrated FPR bounds, downstream-QA noninferiority tests, and regulated-mode claim-safety checks.

Critical strengths
  1. Very rigorous statistical evaluation for a watermarking paper.

  2. The detector-only ablation (Fig. 3a) fixes both the corpus and the answer strings, and the RAG-mechanism comparison matches call budgets and added-token allowances.

Critical weaknesses
  1. The regulated mode is easily defeated, and the theory does not cover this attack. An ordinary 16-call text normalizer, using no keys or oracle information, reduces regulated-mode detection to 0.81% TPR while preserving QA (0.924; Fig. 5). Filter-then-reconstruct reduces it to 0.07%. Theorem 1 holds only under an assumed joint carrier contraction, and given that assumption the proof is a direct substitution.

  2. The headline numbers rest on favorable evaluation settings. The +15.27-point gain over semantic-only comes from a mixed workload in which 25% of the positives use predicate-selective pruning, the attack under which the semantic channel is known to fail (48.27%). The gain thus partly reflects the chosen mixture. The main results also use a 200-document index; at V = 10⁶, 40-call power drops to about 0.81–0.83 (App. E). Calibration costs about 82k calls per profile, must be refit after any configuration change, and assumes the auditor can create known-negative tenants on the target platform.

Questions for authors
  1. For the regulated mode, is there any configuration in which detection survives the ordinary 16-call normalizer at comparable QA? If not, could the abstract and introduction state explicitly that regulated-mode protection holds only against carrier-retaining attacks? I would also like to know what fraction of realistic adversaries you expect to fall outside that class.

  2. Can you report the Fig. 3b comparison (dual-layer vs. semantic-only 40/0 vs. BMCR-only 0/40) separately for each of the four attack families, together with the gain under a uniform reweighting that excludes predicate-selective pruning?

  3. What is the main-comparison performance (Table 1 setting) at index sizes of 10⁵–10⁶ documents with γ ≤ 1%? Which regime do you consider the realistic deployment target?

  4. Theorem 1 is described as a rate-distortion bottleneck. Beyond the finite erasure-channel instance in App. B.2, is there empirical or theoretical evidence that the attacks in your sweep actually satisfy the joint contraction assumption?

  5. In open mode, 0.8% of natural queries surface injected (fabricated) predicates. How should a corpus owner reason about this downstream factual pollution of their own legitimate deployments?

Reviewer_r1iwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a dual-layer watermarking framework for black-box RAG dataset inference, incorporating Bidirectional Masked Contextual Resampling (BMCR), and introduces the RDI benchmark. Experimental results show that the proposed method achieves 99.35% detection power with 40 queries at an empirical false-positive rate of 0.1%.

Critical strengths
  • The paper addresses an important problem of detecting unauthorized dataset usage in black-box RAG systems.

  • The paper provides extensive technical details and comprehensive experimental evaluations across various settings and attack scenarios.

Critical weaknesses
  • The overall presentation and research positioning are difficult to follow, even after multiple careful readings. The paper introduces numerous components, including BMCR, CEC watermarking, adaptive probing, a copula-based detector, and the Thief's Dilemma, without clearly explaining their respective roles or how they relate to the central research problem. Figure 5 contains overlapping text.

  • The methodological novelty relative to existing RAG watermarking approaches is not sufficiently established. WARD already investigates KGW-based lexical watermarking for RAG dataset inference, while RAG-WM and KMW explore knowledge-based watermarking. The proposed framework appears to build substantially on these existing directions by combining semantic and lexical signals and refining the lexical embedding procedure. However, the paper does not clearly explain the fundamental methodological advances beyond these existing approaches.

  • The practical relevance of the theoretical results remains unclear. The Thief's Dilemma relies on a carrier-contraction assumption that links watermark removal to a loss of factual information, but the paper does not establish whether this assumption holds for the proposed watermarking mechanisms. Specifically, the watermarks are designed to preserve the original factual content, making it unclear why removing the watermark signals should necessarily compromise factual utility.

Questions for authors

Could the authors clarify the central contribution relative to existing work? The paper introduces numerous components without sufficiently justifying their necessity, making it difficult to identify the key methodological novelty and understand how each component contributes to the overall framework.

Reviewer_JoVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RAG dataset inference and proposes a dual-layer document watermark. The semantic layer injects long-tail facts in open-domain corpora and, in regulated domains, encodes bits through truth-preserving coordinate orderings and appositive realizations. The lexical layer, BMCR, embeds tokenizer-independent biases in functional words. The auditor probes the two channels with 40 API calls and fuses the evidence with a directional Gaussian-copula test. Experiments on RDI, FARAD and other corpora report 97%–99% detection at a 0.1% false-positive rate.

Critical strengths

The work addresses a practical problem of proving unauthorized use of RAG corpora and motivates the distinction between open domains and regulated domains, where facts cannot be altered. Control experiments with fixed answers and fixed budgets are reasonably thorough and convincingly show that the two channels are complementary under different attacks.

Critical weaknesses
  1. The main paper is highly compressed and uses many self-defined terms (CEC, LMAP+Zoom, OR16, dossier, etc.), and several claims cannot be efficiently evaluated without consulting the extensive appendix. The core method and its limitations should be made more explicit in the main text.

  2. The strongest regulated-mode robustness claim is overstated or insufficiently scoped. The paper does report good results on PubMed and CUAD, but Table 10 shows that, in regulated mode, 16-call graph normalization with fallback leaves QA essentially unchanged (.924 vs. .926) while reducing TPR to 0.81%, and the C16 filtering-to-reconstruction composition reduces it further to 0.07% under the reported utility constraints. The abstract does not mention this. Open-mode robustness also depends on the chosen utility envelope: the OX16 attack reduces detection to 10.48% at a cost of about 4 QA points (.932 to .890), although this point falls outside the paper's stated .02 utility-admitted region.

  3. The theoretical contribution is weak. The proof of Theorem 1 essentially substitutes the evasion constraints into the joint contraction assumed in Definition 1, and this contraction is neither derived nor measured for actual attacks. The constants in Lemma 1 are also not empirically supported.

  4. Novelty is limited. The semantic layer is close to existing knowledge-injection watermarks, and the lexical layer is close to a KGW-style green list. In the slot-matched comparisons that hold the semantic layer fixed and swap only the lexical layer (Table 2, Table 11, Figure 8), BMCR gains only about 2–5 percentage points over the strongest alternative (invariant infilling and its native workflow). The larger gains over weaker adapted baselines such as Active WARD compare full systems and do not isolate the contribution of BMCR itself. In addition, the main benchmark still relies heavily on model-generated RDI data. The human-written and out-of-family experiments are useful, but they are secondary and do not fully replace broader source and deployment diversity.

Questions for authors
  1. Is there a regulated-mode variant that survives C16 or 16-call graph normalization? If not, will the claims about regulated domains in the abstract be revised?

  2. Could you provide full utility–detection curves for the strongest attack families and justify the .02 utility threshold?

  3. Can the contraction coefficients in Definition 1 be estimated for actual attacks? Is Lemma 1 used in the main results?

  4. How does the Table 1 comparison change under the original WARD protocol (content-guided queries, h = 2) and on a human-written corpus such as Wikinews?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.