- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes RelMark. The method embeds private patterns through recommendations between task-equivalent alternatives, and later audits a completed QA dataset by recovering these recommendation directions and testing their alignment with a private key. The paper evaluates the method on CLI, API, and SQL documentation across four generators, and studies rewriting, shuffling, content selection, and targeted editing.
Critical strengths
The paper studies an important problem: auditing source use after retrieval-augmented synthesis when generation logs and live querying are unavailable. The key-blind recovery protocol and independent-key controls are also carefully designed, and the experiments are more thorough than a simple proof-of-concept.
Critical weaknesses
-
RelMark does not robustly establish source provenance. Its signal drops to 0/80 votes under functional-facts synthesis, while useful factual QA is preserved; local editing likewise removes the signal while retaining operational facts. This suggests RelMark traces a particular recommendation rather than actual source use.
-
The setting is highly specialized and intrusive. The median carrier occupies 31.85% of the keyed source, and the method requires verifiably interchangeable alternatives, which largely confines the evaluation to CLI/API/SQL documentation.
-
The baseline comparison is insufficient. Comparing mainly against KGW (Kirchenbauer et al., 2023) does not establish an advance over semantic or source-attribution methods that are explicitly designed to survive rewriting, such as SemStamp (Hou et al., 2024), WASA (Lu et al., 2025), and SemTrace (Zhang et al., 2026). The current result largely shows that semantic content survives semantic transformation better than token bias.
[1] Hou et al., SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation, NAACL 2024. [2] Lu et al., WASA: WAtermark-based Source Attribution for Large Language Model-Generated Data, ACL Findings 2025. [3] Zhang et al., SemTrace: Source-Grounded Semantic Signatures for Tracing LLM Exposure to Protected Documents, 2026.
Questions for authors
- How should RelMark be interpreted as a source-provenance method when the paper itself shows that a benign change in synthesis objective can eliminate the signal while preserving useful source-derived facts?
- Can the authors demonstrate that RelMark remains effective when the carrier occupies a much smaller fraction of the source, and on domains where executable interchangeable alternatives are not naturally available?
- Why is there no direct empirical comparison with semantic watermarking or semantic source-attribution methods, rather than primarily with KGW?