- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SEALBench, a comprehensive benchmark evaluating 11 post-hoc and 4 in-generation image watermarking methods across diverse images and 52 attack conditions. It proposes two new evaluation metrics: Attack-Strength Index (ASI) for comparing attacks based on image distortion and Question Success Rate (QSR) for assessing semantic message recovery. Experiments reveal substantial variations in watermark robustness across methods and attacks, and demonstrate that high bit accuracy does not necessarily guarantee successful semantic message recovery.
Critical strengths
- The paper covers a diverse range of image watermarking methods, image categories, and attack scenarios, providing an extensive evaluation of existing watermarking approaches under a unified benchmark.
- QSR is an interesting attempt to evaluate semantic information recovery beyond traditional bit-level accuracy. This provides an additional perspective on watermark robustness.
- The paper provides detailed descriptions of the benchmark construction process, including data collection, watermark embedding, attack configurations, and evaluation protocols.
Critical weaknesses
- Limited insights beyond empirical evaluations. While the paper presents extensive experiments across diverse watermarking methods and attacks, its key findings are largely descriptive and somewhat expected. The paper lacks deeper analysis to explain these observations or derive actionable insights for developing more robust watermarking methods or new attacking against watermarking.
- Insufficient validation of the proposed ASI metric. Although the paper provides some comparisons at similar mean ASI values, a more systematic evaluation with different attacks calibrated to matched ASI levels would strengthen the analysis. The comparisons with alternative image-quality metrics would also help validate that ASI reliably captures attack-induced distortion across different class of attacks.
- Limited practical relevance of QSR. It implicitly assumes that watermarks encode natural-language messages whose semantic information can be partially recovered. However, practical watermarking and steganography systems often embed structured binary information. Exact bit recovery is essential rather than semantic preservation. To justify the relevance, the authors could raised some examples in the real-world application.
- Lack of qualitative analysis of the proposed metrics. More informative case studies, such as showing representative images under different attacks with similar ASI values or examining recovered messages with different bit accuracy and QSR scores, would help validate the proposed metrics and clarify their practical value.
Questions for authors
- Attack-strength index (ASI) is defined as the LPIPS between an attacked image and the corresponding original image. Based on my understanding, any metrics of image quality of degradation can be used as ASI, such as those used in the watermark attacking competition in NeurIPS 2024 [1]. Could the authors try to have some ablation study with other degradation metrics as ASI, or discuss why they are not proper to be used here?
- Could the authors provide further justification for the design and selection of the 240 messages used for QSR evaluation? How might the choice of messages affect the reliability and generalizability of QSR?
[1] Ding, Mucong, et al. "A Technical Report on “Erasing the Invisible”: The 2024 NeurIPS Competition on Stress Testing Image Watermarks." Advances in Neural Information Processing Systems 38 (2026).
Response to Reviewer QeTC
Thank you for your review and for the concrete suggestions, which we followed. All changes are in the revised PDF.
W1: Limited insights beyond empirical evaluations
Explanation. Attacks with similar distortion differ because they leave different amounts of the watermark signal in the image. For five post-hoc methods, Gaussian noise leaves 93 to 98% of the signal across its strengths, while diffusion regeneration leaves 32 to 42%: noise is added on top of the signal, while regeneration removes a large part of it. Detection follows the retained signal (rank correlation 0.63) more closely than the image distortion (minus 0.40). Detection can also remain after the message is lost: at 1% FPR a method is detected once its bit accuracy exceeds 0.57 to 0.65, while an uncoded message needs every bit to be correct.
For more robust watermarks. (a) Post-hoc methods need to combine two kinds of robustness that now exist in different methods: on real images, VINE keeps 92% detection (TPR@1%FPR) under regeneration but 34% under geometric attacks, and PixelSeal keeps 91% under geometric attacks but 15% under regeneration. (b) In-generation methods need geometric synchronization: all four fall below 50% detection on the same 10 of the 52 conditions, 8 of which contain cropping or rotation. (c) The message needs an error-correcting code: in 105 of the 247 post-hoc method-condition pairs with at least 90% detection, at most half of the messages are recovered exactly, and a BCH code raises the complete-message recovery of PixelSeal and VideoSeal under multiple perturbations (C3) from 0.553 to 0.96.
For new attacks. Diffusion regeneration followed by a mild crop and JPEG compression, which combines the weaknesses in (a) and (b), is the only condition that brings both families below 10% detection on the matched cohort.
In the paper: Section 5 (last paragraph), Section 5.2, Appendix F.10 (Tables 38 and 39).
W2: Insufficient validation of the ASI metric
We did both. At the same ASI (0.20, interpolated between measured strengths), mean detection of post-hoc methods on real images is 98.5% under Gaussian noise but 27.6% under diffusion regeneration (Table 32). With eight other image-quality metrics in place of LPIPS, the finding holds: every metric finds pairs of attack conditions in which the less distorted one is detected more than 50 percentage points less often (30 pairs under ASI, 3 to 62 under the others, Table 33).
In the paper: Section 4 C, Section 5.2, Appendix F.8.
Q1: Ablation with other degradation metrics
We added eight metrics: LPIPS-VGG, DISTS, SSIM, MS-SSIM, RMSE, NMI, CLIP distance, and the four per-image terms of the NeurIPS 2024 competition score (PSNR, SSIM, NMI and LPIPS, with their published weights). The main finding holds under each of them, and the method ranking changes little (Kendall tau 0.72 to 1.00). The other four terms (FID, CLIP-FID, aesthetics and artifacts) are left out because they are not distances between an attacked image and its own original.
In the paper: Appendix F.8.
W3: Limited practical relevance of QSR
QSR is reported next to complete-message recovery, not instead of it. Each question asks for one field of the message, so QSR counts how many fields are recovered. This matters when a watermark carries provenance fields as text, such as the creator or the generating model, a possible form of the machine-readable marking that the EU AI Act requires. Under the four image-sharing chains (resizing and JPEG compression), PixelSeal and VideoSeal recover 49% and 54% of the messages exactly, but their QSR is 0.78 and 0.83. Under the three removal attacks of the main suite it is below 0.01.
In the paper: Section 4 B, Section 5.5, Appendix F.7.
W4: Lack of qualitative analysis
We added both. Figure 5 shows PixelSeal images under two pairs of attacks. In each pair, the attack with the lower ASI removes the watermark and the attack with the higher ASI does not, which is a stricter test than similar ASI. The outcomes shown hold for 1,116 of the 1,193 images of the matched synthetic cohort. Table 37 shows recovered messages with their bit accuracy and QSR.
In the paper: Appendix F.9 (Figure 5, Table 37).
Q2: Design and selection of the 240 messages
The messages are built from fixed templates with randomly chosen words and fictional people, so no answer can be guessed from world knowledge, and each answer is a single word of the message.
To test whether QSR depends on the chosen messages, we recomputed it on 2,000 bootstrap resamples of the messages, using the scores already measured. The QSR of every method stayed within 0.04 of its reported value, and for 43 of the 45 pairs among the 10 message-carrying methods, the method with the higher QSR stayed the same in at least 95% of the resamples. The other two pairs, VINE against VideoSeal and against PixelSeal, differ in QSR by 0.010 and 0.025.
In the paper: Appendix D, Appendix F.5.