- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies post-hoc watermarking, which adds a watermark to text that already exists. Sparse-editing schemes replace a small portion of the tokens, but each edit carries only a weak signal. Full-rewriting schemes regenerate the passage with a watermarked LLM. They carry a strong signal, but they are slow and can change the text in arbitrary ways instead of modifying only a few tokens. The paper proposes ReDiffMark, which follows the sparse-editing idea but uses a discrete (masked) diffusion language model to make the edits. A secret key selects the positions to replace, and these positions are masked. For each masked position, the diffusion model gives a distribution over tokens, and key-seeded Gumbel-max sampling selects the token that carries the watermark. A position is changed only where the model is uncertain enough. The detector re-runs the same steps on the received text and tests the keyed scores against a calibrated null. On clean 200-token news passages, the authors report detection close to that of full-rewriting schemes (TPR at 1% FPR of 0.94, against 0.95 to 1.00) while changing about 8% of tokens instead of 65–78%, at a much lower latency.
Critical strengths
-
I think designing more efficient post-hoc watermarking is an important research question, with many applications. The method reaches a useful operating point: on clean news passages, TPR@1% is 0.94 while about 8% of tokens change, at 0.37 s per passage (Table 1).
-
The paper includes sufficient baselines, covering both sparse editing and full rewriting, and the comparison between them is quite fair. The controlled ablation of the scoring rule is also informative.
Critical weaknesses
1. Apart from changing fewer tokens, the benefit over full rewriting is not obvious.
- The clear gain is that far fewer tokens change. But one of the paper's main arguments against rewriting is its cost. The proposed method does embed faster, but its total compute is similar to that of the rewriting baselines, and its detection is slower than for most of them, because it must re-run the diffusion model (Table 11). The fast key-only detector loses much of the detection power. Overall, the cost remains similar.
- The claim to outperform all baselines rests on the latency term of the composite score. Without that term, PostMark ranks first.
- The method is also less robust than full rewriting, under both word substitution and back-translation, although it is much more robust than the sparse-editing baselines.
- The diffusion model is not what carries the signal: a RoBERTa infiller gives about the same detection, though with less fluent edits (Table 17). The method is therefore close to earlier sparse substitution schemes (Yang et al., 2022; Yoo et al., 2023), with Gumbel sampling (Kuditipudi et al., 2024) as the main change.
2. The fidelity metrics do not show that the watermarked text stays useful.
- The paper does measure semantic similarity, but with a sentence-embedding score averaged over passages. Such scores can stay high when a single word changes the meaning, such as a negation or a modal verb. The paper's own first example, a health-advice passage, changes "will help" to "may help", the same kind of edit the paper criticizes in a baseline. The paper also notes that edits may change nuance.
- This matters more here because the signal depends on edits that an external language model finds unlikely: when a verifier keeps only the edits that barely change perplexity, detection largely collapses (Table 6). The edits that carry the watermark are the ones most likely to shift meaning.
- The right test is whether the watermarked text still serves its purpose. For example, when post-hoc watermarks are used to detect benchmark contamination, models should rank the same on the original and the watermarked benchmark. For RAG documents, a system should still give the same answers. The paper reports no such evaluation.
Questions for authors
-
Does the watermarked text stay useful for its intended purpose? For example, if a benchmark is watermarked post hoc to detect contamination, do models keep the same accuracy and ranking on the watermarked version? For RAG documents, do the answers stay correct? If utility is preserved, my concern about the fidelity metrics is resolved.
-
Could you report results for a targeted removal attack and for spoofing? For uses such as contamination detection, adversarial robustness may matter less, but readers still need to know the limits. For example, an attacker could flag the tokens that an external language model finds least likely and refill them without the key. Or the attacker could collect many watermarked passages with their originals and learn which tokens the key favors at each position, as in watermark stealing (Jovanović et al., 2024).
-
How do embedding and detection costs compare with the rewriting baselines under batched serving? The composite score uses single-passage latency, which favors a two-pass method, while detection requires re-running the diffusion model. This comparison would show whether the cost advantage holds in practice.
minor:
- The default position keys break after a single insertion or deletion, which conflicts with the goal of surviving light edits. Seeding keys from a wider context of unedited tokens would avoid this. The detector should then score each repeated (context, token) pair only once, to keep the p-values valid (Fernandez et al., 2023).
- Please state whether detection runs on decoded strings re-tokenized by the diffusion model or on stored token ids, and report detection after word-level edits on decoded text. With position keys, a single tokenization mismatch shifts all later indices.
- The RAG-attribution use case needs detection after an LLM rewrites the passage. After one paraphrase, every compared scheme, this one included, loses most of its signal. I would state the claim as attribution of verbatim or lightly edited copies.
Kind Request for Your Feedback
Dear Reviewer 8vTg,
Thank you for your constructive feedback. We believe we have addressed your concerns to a large extent in our response.
In short, we added downstream utility results-RAG QA and retrieval stability-to resolve your fidelity concern. We also added batched-serving costs on efficiency concerns, and the targeted-removal and watermark-stealing experiments you asked for.
Given that the deadline is approaching, could you please take a moment to review our updates? If you have any further questions, we would be more than happy to provide additional clarification.
Thank you again for your time and effort in reviewing our paper. We hope these updates help clarify the manuscript.
Best regards,
The Authors