- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a dataset watermarking method that increases the co-occurrence of randomly selected word pairs through meaning-preserving local edits. It enables black-box detection using generated text and provides a hypergeometric test with provable false-positive control. Experiments on four language models and three benchmarks demonstrate effective detection under fine-tuning, data dilution, and lightweight text modifications, while largely preserving dataset utility.
Critical strengths
The paper provides a simple, API-compatible dataset watermarking framework that enables black-box detection without access to model logits or token-level generation controls. The randomization-based statistical test provides valid false-positive control under its stated assumptions, and experiments demonstrate promising detection performance across multiple models and datasets.
Critical weaknesses
- Limited novelty of the watermarking mechanism. The proposed method relies on a relatively straightforward mechanism: artificially increasing the co-occurrence of randomly selected word pairs through meaning-preserving lexical edits. Similar association-based watermarking mechanisms have been extensively explored in code dataset watermarking. More recent work, such as PuzzleMark [2], has moved beyond conventional co-occurrence patterns toward more sophisticated and stealthy watermark constructions. The proposed method primarily adapts a simple statistical association mechanism to natural language, while its false-positive guarantee follows from a standard hypergeometric randomization test. Overall, the methodological novelty appears limited.
- Insufficient evaluation of watermark stealthiness and robustness. Prior work, including DeCoMa [1], has demonstrated that artificially introduced trigger–target associations in code datasets can leave detectable statistical artifacts and be removed through data-level analysis. PuzzleMark [2] further highlights the vulnerability of conventional co-occurrence-based watermarks and proposes a more robust alternative. Although these studies focus on code datasets, the proposed method similarly relies on artificially amplified lexical associations. However, the paper does not evaluate whether its watermark can be identified and removed through statistical analysis without knowledge of the secret key. This omission raises substantial concerns about the watermark's stealthiness and practical robustness. References [1] DeCoMa (ISSTA 2025) Xiao, Y., et al. DeCoMa: Detecting and Purifying Code Dataset Watermarks through Dual Channel Code Abstractions. Proceedings of the ACM on Software Engineering, ISSTA 2025. https://doi.org/10.1145/3728952 [2] PuzzleMark (FSE 2026) Huang, H., et al. PuzzleMark: Implicit Jigsaw Learning for Robust Code Dataset Watermarking in Neural Code Completion Models. Proceedings of the ACM on Software Engineering, FSE 2026. https://doi.org/10.1145/3797122
Questions for authors
- Novelty: How does the proposed word-pair co-occurrence watermark fundamentally differ from existing trigger–target association-based dataset watermarking methods? What is the key technical innovation beyond meaning-preserving lexical edits and a standard hypergeometric test?
- Detectability and Removal: Prior work such as DeCoMa has demonstrated that anomalous trigger–target associations in code datasets can be identified and removed through statistical analysis. Although DeCoMa does not currently provide a docstring → docstring detection mode, its existing implementation already supports token-level analysis of docstrings, making such an extension relatively straightforward. Given that the proposed watermark similarly relies on artificially amplified word-pair co-occurrences, I am particularly interested in whether it would survive a DeCoMa-style docstring → docstring detection and removal procedure without access to the secret key. Could the authors evaluate this threat and report the watermark detection rate after data purification?
Dear Reviewer ZaHq,
Thank you again for your constructive feedback and review. As today is the last day of the discussion period, we wanted to check whether our responses have addressed your concerns. In particular, we ran additional experiments and show that our method remains effective on an additional instruction-following dataset (Databricks Dolly 15K), and independently keyed watermarks remain reliably detectable when two watermarked datasets are used together for fine-tuning. We also ran an ablation controlling for the rephrasing model.
If anything remains unclear, we would be happy to provide further details. If our responses have resolved your concerns, we would greatly appreciate it if you would consider updating your score. Thank you for your time and consideration.
Best regards, The Authors