- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to detect a multi-bit watermark before trying to recover its embedded message. PIPER uses a two-level vocabulary partition so that the same tokens support both tasks. It first tests watermark presence using an exact binomial test over distinct contexts, then recovers the message through voting and error correction. The paper proves false-positive control under a random-key model with an idealized pseudorandom function, and evaluates detection, message recovery, text quality, and robustness across several models and datasets.
Critical strengths
-
The paper addresses a useful problem. Searching over possible messages can increase false positives if the full search procedure is not properly calibrated. PIPER provides a clear way to keep the presence decision independent of this search.
-
The method has a clear statistical basis. The two-level partition lets the same tokens support both detection and message recovery. Under the stated random-key assumptions, context deduplication supports an exact binomial test whose false-positive guarantee does not depend on the decoder.
-
The experiments show strong detection performance and examine its limits. At 300 tokens and a nominal 1% false-positive rate, the 8-bit setting achieves 98.7% detection and 95.0% correct acceptance and message recovery. Additional tests cover different models, text domains, fixed keys, and text edits.
Critical weaknesses
-
Reliability for a fixed deployment key remains uncertain. The guarantee averages over random keys, while Table 4 reports an OpenGen false-positive rate of 0.333% at a target of 0.1% for one fixed key. Although the paper acknowledges this limitation, it matters for applications that keep the same key.
-
The gain from shared evidence becomes smaller when text quality is considered. The main comparison fixes the generation bias, while the roughly quality-matched results in Table 13 show only a small CAR gain at 100 tokens. At 200 and 300 tokens, the tuned 25/75 disjoint baseline has higher CAR, although it also has slightly higher perplexity.
-
Message recovery is weak after substantial text changes. Table 15 reports CAR of only 35.9% after mixing equal amounts of watermarked and unwatermarked text, and 10.3% after T5 paraphrasing. These results limit the method's value for recovering source information from edited text, even when watermark presence remains detectable.
Questions for authors
-
Reliability across fixed keys. How much does the false-positive rate vary across individual keys, particularly on OpenGen? How often does it exceed the target level?
-
Comparison at similar error rates and text quality. Do the gains over CTWL and tuned disjoint allocation remain when both the actual false-positive rate and text quality are matched? Please explain the small or reversed CAR differences in Table 13.
-
Message recovery after edits. Can you report the stronger editing and mixing settings already evaluated but omitted from the tables, and compare message recovery with a strong baseline under the same attacks?
4 comments
Comparisons at Matched False-Positive Rates and Text Quality (W2, Q2)
To more clearly demonstrate the advantages of PIPER over CTWL and Disjoint under comparable text quality, we conducted additional experiments with (T=300). Admittedly, while PIPER achieves strong performance in watermark presence detection, its message decoding accuracy is somewhat lower. To address this limitation, we further improved the decoding mechanism while preserving our two-stage detect-then-decode framework. Specifically, the original approach randomly partitions the vocabulary into four disjoint subsets and increases the probabilities of tokens in the subset corresponding to the target symbol. In contrast, our improved approach employs two independent random permutations, each excluding a designated partition, and applies a positive bias only to the intersection of the two remaining token sets. This modification enables a minimum-based selection among candidate symbols at each watermark position during decoding, effectively improving message recovery accuracy for short texts.
We note that CTWL faces a significant scalability limitation with increasing payload length, since its decoding procedure requires exhaustive evaluation of all possible candidate messages. For a 32-bit payload, the candidate space grows to ,making exhaustive decoding computationally impractical. In contrast, PIPER avoids such exhaustive message enumeration. Furthermore, as the text length increases, more decoding evidence can be accumulated for each payload position,which is expected to substantially improve PIPER’s message recovery accuracy. The following table presents the detailed experimental results under comparable text quality.
Table: Performance comparison for 8-bit and 16-bit payloads under comparable text quality.
| Method | Bias δ | 8-bit PPL ↓ | 8-bit TPR (%) ↑ | 8-bit CAR (%) ↑ | 16-bit PPL ↓ | 16-bit TPR (%) ↑ | 16-bit CAR (%) ↑ |
|---|---|---|---|---|---|---|---|
| PIPER | 2.0 | 8.82 | 98.70 | 95.00 | 8.78 | 99.00 | 77.50 |
| PIPER (Improved) | 2.0 | 8.7214 | 98.70 | 97.90 | 8.7562 | 98.90 | 93.60 |
| CTWL | 3.0 | 8.6886 | 98.20 | 98.20 | 8.7979 | 98.10 | 98.10 |
| Disjoint 25/75 | 2.2 | 8.7114 | 97.70 | 94.60 | 8.9431 | 97.67 | 86.33 |
| Disjoint 50/50 | 2.2 | 8.8592 | 99.00 | 89.80 | 8.7257 | 98.67 | 60.33 |
| Disjoint 75/25 | 2.2 | 8.8768 | 99.20 | 52.20 | 9.0031 | 100.00 | 16.67 |
explain the small or reversed CAR differences in Table 13: This is primarily because PIPER can utilize only half of the original vocabulary for payload embedding, whereas Disjoint 25/75 can leverage the entire vocabulary to embed payload information at 75% of token positions. Consequently, PIPER accumulates less effective evidence for payload decoding, which partly explains its lower message recovery accuracy. Nevertheless, our subsequent improvements to PIPER effectively mitigate this limitation, achieving better decoding performance than Disjoint 25/75 in our experiments.
Payload Recovery under Text Editing and Mixing (W3, Q3)
Thank you for asking us to report stronger attack settings and compare message recovery with other multi-bit watermarks. PIPER is designed to support multi-bit attribution while controlling the false-positive rate of watermark detection. Editing and paraphrasing can alter the tokens and local contexts that carry watermark evidence, and mixing reduces the amount of watermarked text available to the detector and decoder.
The following table compares the performance of PIPER and other methods under identical attack settings. PIPER demonstrates strong watermark detection performance under most attacks. However, under more challenging attacks, such as T5-based paraphrasing, 30% token replacement, and truncation from 300 to 100 tokens, existing multi-bit watermarking methods generally suffer substantial performance degradation. Improving robustness against such severe attacks remains an important direction for future work. In particular, we aim to build upon our two-stage detect-then-decode framework to develop a more robust multi-bit watermarking scheme.
Table: Robustness comparison under different attacks. Each entry reports TPR@1% FPR / CAR (both in percentages). CAR denotes the joint correct acceptance-and-recovery rate. The best TPR and CAR for each attack setting are highlighted separately in bold.
| Attack | Setting | PIPER | MPAC | BREW | CTWL |
|---|---|---|---|---|---|
| Clean | No attack | 99.50 / 97.28 | 97.00 / 90.00 | 63.33 / 63.33 | 95.67 / 95.67 |
| Token-preserving synonym edits | 5% | 99.00 / 90.70 | 93.67 / 80.67 | 48.33 / 48.33 | 92.33 / 92.33 |
| Token-preserving synonym edits | 10% | 96.80 / 72.10 | 91.00 / 74.00 | 34.67 / 34.67 | 81.33 / 81.33 |
| Token-preserving synonym edits | 20% | 67.80 / 17.00 | 69.33 / 42.00 | 13.67 / 13.67 | 49.00 / 49.00 |
| Token-preserving synonym edits | 30% | 26.70 / 1.90 | 36.33 / 15.67 | 4.33 / 4.33 | 19.33 / 19.00 |
| Token-increasing synonym edits | 5% | 99.00 / 89.10 | 92.67 / 81.00 | 0.00 / 0.00 | 91.67 / 91.67 |
| Mixing | 50% watermarked | 83.60 / 35.90 | 26.33 / 10.00 | 18.67 / 18.67 | 19.33 / 19.33 |
| Truncation | 200 tokens retained | 99.10 / 90.10 | 87.33 / 71.33 | 59.67 / 59.67 | 82.33 / 82.33 |
| Truncation | 100 tokens retained | 93.00 / 48.20 | 26.33 / 20.00 | 96.33 / 41.00 | 4.33 / 4.33 |
| Paraphrasing | T5 | 59.70 / 10.30 | 13.67 / 8.67 | 81.00 / 4.00 | 1.67 / 1.67 |
Thank you for the detailed response. The new results address much of my concern about fixed-key false positives. I still have concerns about whether the gains hold at comparable false-positive rates and how reliably the method recovers messages after substantial edits. I am keeping my original rating.
False-Positive Rates Across Fixed Deployment Keys (W1, Q1)
As you correctly pointed out, our current theoretical analysis is conducted under the random-key setting, whereas practical deployment typically relies on a fixed key. This assumption is also commonly adopted in existing multi-bit watermarking methods. Extending our theoretical guarantees to the fixed-key setting is an important direction for future work. Interestingly, our additional experiments comparing a single fixed key with 100 independently sampled keys demonstrate that PIPER maintains reliable performance even under the fixed-key setting. The detailed results are presented in the following table.
Table: Fixed-key false-positive rates on large evaluation sets.