- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates practical black-box spoofing attacks against LLM watermarks and proposes RLSpoofer, a sample-efficient spoofing method. Based on a theoretical analysis suggesting that low-dimensional watermark-induced probability shifts can be learned more efficiently than the full distribution, the authors propose selectively reconstructing the remaining watermark residuals. To this end, RLSpoofer employs GRPO. Using only 100 training pairs, RLSpoofer achieves substantial improvements over existing baselines across multiple watermarking schemes.
Critical strengths
-
Investigating black-box watermark spoofing attacks is important for understanding the practical security risks and vulnerabilities of LLM watermarking systems.
-
The proposed method achieves substantial improvements over existing baselines, particularly against distortion-free watermarking schemes.
Critical weaknesses
-
The theoretical formulation measures watermark-induced probability shifts under the same context, whereas the implementation estimates them using a reference model conditioned on different paraphrases. Since semantic equivalence does not imply similar next-token distributions, the estimated residual may capture generic paraphrasing variations rather than watermark-specific shifts. Further justification would help clarify whether the proposed residual estimation captures watermark-specific information.
-
The evaluation primarily relies on a P-SP threshold of 0.7, which may be insufficient to capture fine-grained semantic inconsistencies in relatively long texts of 500 tokens. A high similarity score does not necessarily guarantee semantic preservation, particularly for longer texts where subtle differences in details may be obscured by overall semantic similarity. This is a known limitation of cosine similarity-based metrics, and the chosen threshold of 0.7 appears relatively lenient. Moreover, RLSpoofer achieves substantially lower LLM-judge scores than the baselines in several settings. These results raise concerns that the improved spoofing performance may come at the cost of semantic fidelity and output quality, weakening the claim of generating semantically coherent text.
-
Despite emphasizing effectiveness against distortion-free watermarks, the evaluation only includes PF-Watermark and PMark, omitting the widely studied EXP watermark. Evaluating against EXP would provide stronger evidence of generalizability across different distortion-free watermarking mechanisms.
-
On PF-Watermark with Qwen3-4B, SSR increases from 20.8% (50 samples) to 62.0% (100 samples), but drops sharply to 25.3% (200 samples) (Table 7, p. 8). This non-monotonic behavior raises concerns about the reliability of the headline result.
Questions for authors
Please see the four critical weaknesses above. Additionally, I have a question regarding cross-watermark transferability.
- Transferability between similar logit-based watermarks is understandable, as they share similar green-list mechanisms and the same hash key in the experimental setup. However, the substantial transferability between PF and EWD is difficult to understand, particularly given its asymmetric results (PF → EWD: 50.5% vs. EWD → PF: 7.75%). Could the authors provide further analysis or intuition to explain this asymmetric transferability across different watermarking mechanisms?
7 comments
Q4. How stable are the PF-Watermark results across different training-set sizes?
We thank the reviewer for the valuable comments. To assess the stability of PF-Watermark results, we evaluated two additional random seeds and training subsets for each data size using Qwen3-4B, yielding six additional runs. As shown below, the 100-pair setting consistently achieves 60.5–63.5% SSR, confirming the reproducibility of our results. We hypothesize that the non-monotonic trend arises from PF's distortion-free property in expectation, which may cause watermark-specific distributional shifts to become harder to capture as the dataset grows. We have included these results and discussed this potential explanation in the Appendix C.6 of our revised manuscript.
| Method | 50 | 100 | 200 |
|---|---|---|---|
| Untrained | 3.0 | 3.0 | 3.0 |
| Seed 1 | 20.8 | 62.0 | 25.3 |
| Seed 2 | 22.5 | 60.5 | 24.8 |
| Seed 3 | 21.5 | 63.5 | 27.0 |
| Mean ± Std | 21.6 ± 0.9 | 62.0 ± 1.5 | 25.7 ± 1.2 |
Q5. What explains the asymmetric cross-watermark transferability between PF and EWD?
A: We thank the reviewer for the valuable comments. We hypothesize that the asymmetric transfer arises from differences in watermarking mechanisms and detector statistics. In particular, PF relies on key-dependent sampling correlations, whereas EWD uses entropy-weighted token statistics. We additionally evaluated genuine watermarked texts and found cross-detection rates of only 0% (PF->EWD) and 2% (EWD->PF). This suggests that the observed transfer is not simply inherited from the original watermark signals, but may arise from detector-specific statistical patterns induced during attack optimization. We have clarified this potential explanation in the Appendix C8 of the revised manuscript.
| Watermarked Text | EWD Detector | PF Detector |
|---|---|---|
| EWD | 100% | 2% |
| PF | 0% | 100% |
We sincerely thank you again for your valuable time and constructive feedback, which have greatly improved the quality and clarity of our manuscript.
Gentle Follow-up on the Rebuttal
Dear Reviewer QC2w,
We sincerely appreciate your thoughtful review, valuable suggestions, and the time and effort you have devoted to evaluating our submission. We have provided additional experiments and clarifications in response to your valuable feedback.
We hope our responses and additional experimental results could address your concerns. We would be very grateful if you could share any further feedback or remaining concerns at your convenience. We would be happy to provide additional clarifications if needed.
Best regards,
Paper 11130 Authors
We sincerely thank you for your valuable comments. We hope our response below could address your concerns.
A: We thank the reviewer for the valuable comment. We acknowledge that our practical surrogate does not exactly recover the theoretical watermark-induced residual, as the true distributions are inaccessible under our black-box setting. Since watermark signals can persist after weak paraphrasing [1], we adopt a distributional view, using a lightweight reference model to approximate these shifts.
Furthermore, to validate this approximation, we conducted additional token-level analysis on EWD using 100 paired samples and 9,600 sampled tokens. Specifically, we compared the initial surrogate residual estimated by Llama-3.2-3B with the actual watermark-induced positive probability shifts computed by Llama-3.1-8B.
These results show statistically supported positive alignment with actual watermark-induced shifts, exceeding the unwatermarked generation. While contextual variations may still contribute, our surrogate captures meaningful watermark-related signals. We have included this analysis and clarified its limitations in the Appendix C.10 revised manuscript.
[1] Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models ACL2024
A: We thank the reviewer for the valuable question. Prior studies have shown that watermark-induced distributional perturbations can introduce trade-offs between detectability and text quality, including semantic consistency [2]. Similarly, reconstructing watermark-related signals may compromise semantic fidelity. We therefore acknowledge this trade-off and evaluate Qwen3-4B under stricter P-SP thresholds of 0.80 and 0.90. As shown below, RLSpoofer consistently outperforms all baselines. We have included these results in Appendix C.1 and explicitly discussed the semantic fidelity trade-off in the Limitations section of our revised manuscript.
[2] Context-aware Watermark with Semantic Balanced Green-red Lists for Large Language Models. EMNLP2024
A: We thank the reviewer for the valuable comment. We additionally evaluated RLSpoofer on the distortion-free EXP watermark, with results shown below. RLSpoofer achieves 40.5% SSR with only 100 training pairs, significantly outperforming Distill. These results further support the effectiveness of RLSpoofer across different distortion-free watermarking mechanisms. We have included these additional results in the revised manuscript.