- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- PAMark is a lightweight multi-bit watermarking scheme that combines p-ary (full-payload) encoding with Gumbel-ranking-based token selection.
- During generation, each token is designed to provide evidence for the entire payload, and candidate vectors are ranked using Gumbel noise and model probabilities to select the token.
- During decoding, the payload is reconstructed through coordinate-wise voting without requiring LM logits (model-free recovery).
Critical strengths
The topic of robust multi-bit watermarking for LLM-generated text is interesting and important.
Critical weaknesses
- Although BiMark and MPAC are included as baselines in Table 2, they are excluded from the attack evaluation in Table 3. As a result, the number of baselines decreases from four to two, making the experimental comparison less comprehensive.
- The main paper reports only Exact Recovery under attack conditions, which is insufficient for a comprehensive evaluation of watermark detection performance. Although Table 9 provides additional metrics such as TPR and AUC, FPR results are still missing. Reporting FPR would help provide a more complete assessment of the proposed method.
- Although Binomial shows lower Exact Recovery in Table 3 and Figure 2, it outperforms PAMark on the other reported metrics in Table 9. This raises concerns about the advantages of PAMark over existing methods, as the experimental results do not clearly demonstrate its superiority.
- The motivation for selecting Exact Recovery as the primary evaluation metric is not sufficiently justified. High Exact Recovery alone does not necessarily indicate an effective watermarking scheme, particularly if it is accompanied by a high false positive rate (FPR).
- Table 3 and Figure 2 show that Exact Recovery under clean conditions is approximately 80%, which seems relatively low even in the absence of attacks. Since Exact Recovery is already limited under clean conditions, further degradation under attacks is expected, raising concerns about the practical effectiveness of the proposed method.
- Although some experiments use 200 prompts, most evaluations appear to be conducted using only 60 prompts. The inconsistent number of prompts across experiments makes it difficult to interpret the results and compare performance across different settings.
Questions for authors
- Could the authors provide additional results under paraphrasing attacks, such as DIPPER? To evaluate watermark robustness more comprehensively, it would be helpful to report Exact Recovery along with detection performance at low false positive rates, such as TPR@1% FPR and TPR@5% FPR.
- Most attack evaluations appear to use only 60 prompts, which may be insufficient to provide reliable experimental results. Could the authors expand this evaluation to 200 prompts to improve the reliability of the results? If expansion is difficult, please provide the confidence intervals for the 60 prompt results.
- Although TPR and AUC results are provided for Syn-10 and Syn-30, could the authors also report FPR to provide a more complete assessment of detection performance?