- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies RAG dataset inference: deciding from black-box answers whether a suspect RAG service uses a protected corpus. It proposes a dual-layer watermark. The semantic layer injects long-tail fact bindings in open domains, or uses truth-preserving reorderings in regulated domains. The lexical layer, BMCR, biases functional words in a tokenizer-independent way. A 40-call Prober-Verifier fuses the two channels with a Gaussian-copula test at α = 10⁻³. The theory gives a "Thief's Dilemma" bound under an assumed joint carrier contraction, plus query-budget certificates. Experiments are mainly on RDI, a new LLM-generated benchmark, with FARAD and Wikinews as external tests. The method reaches 97.1% [email protected]% FPR on RDI-Hard, at least 98.17% detection across a 150-setting attack sweep, and a 15.27-point gain over semantic-only on a mixed-attack workload. The paper also reports calibrated FPR bounds, downstream-QA noninferiority tests, and regulated-mode claim-safety checks.
Critical strengths
-
Very rigorous statistical evaluation for a watermarking paper.
-
The detector-only ablation (Fig. 3a) fixes both the corpus and the answer strings, and the RAG-mechanism comparison matches call budgets and added-token allowances.
Critical weaknesses
-
The regulated mode is easily defeated, and the theory does not cover this attack. An ordinary 16-call text normalizer, using no keys or oracle information, reduces regulated-mode detection to 0.81% TPR while preserving QA (0.924; Fig. 5). Filter-then-reconstruct reduces it to 0.07%. Theorem 1 holds only under an assumed joint carrier contraction, and given that assumption the proof is a direct substitution.
-
The headline numbers rest on favorable evaluation settings. The +15.27-point gain over semantic-only comes from a mixed workload in which 25% of the positives use predicate-selective pruning, the attack under which the semantic channel is known to fail (48.27%). The gain thus partly reflects the chosen mixture. The main results also use a 200-document index; at V = 10⁶, 40-call power drops to about 0.81–0.83 (App. E). Calibration costs about 82k calls per profile, must be refit after any configuration change, and assumes the auditor can create known-negative tenants on the target platform.
Questions for authors
-
For the regulated mode, is there any configuration in which detection survives the ordinary 16-call normalizer at comparable QA? If not, could the abstract and introduction state explicitly that regulated-mode protection holds only against carrier-retaining attacks? I would also like to know what fraction of realistic adversaries you expect to fall outside that class.
-
Can you report the Fig. 3b comparison (dual-layer vs. semantic-only 40/0 vs. BMCR-only 0/40) separately for each of the four attack families, together with the gain under a uniform reweighting that excludes predicate-selective pruning?
-
What is the main-comparison performance (Table 1 setting) at index sizes of 10⁵–10⁶ documents with γ ≤ 1%? Which regime do you consider the realistic deployment target?
-
Theorem 1 is described as a rate-distortion bottleneck. Beyond the finite erasure-channel instance in App. B.2, is there empirical or theoretical evidence that the attacks in your sweep actually satisfy the joint contraction assumption?
-
In open mode, 0.8% of natural queries surface injected (fabricated) predicates. How should a corpus owner reason about this downstream factual pollution of their own legitimate deployments?