acceptodds
Under review as a conference paper at ICLR 2027

SEALBench: A Comprehensive Benchmark for Image Watermark Detection, Robustness, and Recovery

Abstract

As concerns over the misuse of Generative AI models continue to grow, watermarking has emerged as a promising solution, embedding a detectable signal into generated or existing images. Existing image watermarking benchmarks have three limitations: 1) inconsistency across data for comparing in-generation and post-hoc watermarking methods, 2) uneven attack settings for image/watermark distortion, and 3) bit accuracy that does not measure whether the decoded message retains its meaning. To this end, we present SEALBENCH, an image watermarking benchmark that evaluates 4 in-generation and 11 post-hoc watermarking methods on 25,864 real, 15,000 manipulated, and 12,665 synthetic images. We further evaluate image-level watermarking attacks in four attack categories, from single image transformations to editing or watermark removal followed by additional processing, and propose the attack-strength index (ASI) to unify all image-level watermarking attacks on a single damage axis. We evaluate and propose Question Success Rate (QSR), i.e. embedded message recovery, by testing whether a frozen language model can accurately answer questions using the extracted text. Our evaluation of State-of-the-art methods on SEALBENCH reveal interesting findings: a) both in-generation and post-hoc methods remain vulnerable to attacks, although their robustness varies significantly across attack types, b) attacks that produce similar image distortion can have different effects on watermark detection and c) High bit accuracy does not necessarily imply complete message recovery or preservation of the message’s meaning. We believe SEALBENCH will support more consistent evaluation of image watermark robustness and encourage research on methods that preserve both watermark detection and message recovery under different attack settings.

Primary area:
datasets and benchmarks
Keywords:
Image watermarking, robustness evaluation, benchmarking, generative models, image provenance, message recovery

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QeTCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SEALBench, a comprehensive benchmark evaluating 11 post-hoc and 4 in-generation image watermarking methods across diverse images and 52 attack conditions. It proposes two new evaluation metrics: Attack-Strength Index (ASI) for comparing attacks based on image distortion and Question Success Rate (QSR) for assessing semantic message recovery. Experiments reveal substantial variations in watermark robustness across methods and attacks, and demonstrate that high bit accuracy does not necessarily guarantee successful semantic message recovery.

Critical strengths
  1. The paper covers a diverse range of image watermarking methods, image categories, and attack scenarios, providing an extensive evaluation of existing watermarking approaches under a unified benchmark.
  2. QSR is an interesting attempt to evaluate semantic information recovery beyond traditional bit-level accuracy. This provides an additional perspective on watermark robustness.
  3. The paper provides detailed descriptions of the benchmark construction process, including data collection, watermark embedding, attack configurations, and evaluation protocols.
Critical weaknesses
  1. Limited insights beyond empirical evaluations. While the paper presents extensive experiments across diverse watermarking methods and attacks, its key findings are largely descriptive and somewhat expected. The paper lacks deeper analysis to explain these observations or derive actionable insights for developing more robust watermarking methods or new attacking against watermarking.
  2. Insufficient validation of the proposed ASI metric. Although the paper provides some comparisons at similar mean ASI values, a more systematic evaluation with different attacks calibrated to matched ASI levels would strengthen the analysis. The comparisons with alternative image-quality metrics would also help validate that ASI reliably captures attack-induced distortion across different class of attacks.
  3. Limited practical relevance of QSR. It implicitly assumes that watermarks encode natural-language messages whose semantic information can be partially recovered. However, practical watermarking and steganography systems often embed structured binary information. Exact bit recovery is essential rather than semantic preservation. To justify the relevance, the authors could raised some examples in the real-world application.
  4. Lack of qualitative analysis of the proposed metrics. More informative case studies, such as showing representative images under different attacks with similar ASI values or examining recovered messages with different bit accuracy and QSR scores, would help validate the proposed metrics and clarify their practical value.
Questions for authors
  1. Attack-strength index (ASI) is defined as the LPIPS between an attacked image and the corresponding original image. Based on my understanding, any metrics of image quality of degradation can be used as ASI, such as those used in the watermark attacking competition in NeurIPS 2024 [1]. Could the authors try to have some ablation study with other degradation metrics as ASI, or discuss why they are not proper to be used here?
  2. Could the authors provide further justification for the design and selection of the 240 messages used for QSR evaluation? How might the choice of messages affect the reliability and generalizability of QSR?

[1] Ding, Mucong, et al. "A Technical Report on “Erasing the Invisible”: The 2024 NeurIPS Competition on Stress Testing Image Watermarks." Advances in Neural Information Processing Systems 38 (2026).

1 comment
AuthorsOpenReview ↗

Response to Reviewer QeTC

Thank you for your review and for the concrete suggestions, which we followed. All changes are in the revised PDF.

W1: Limited insights beyond empirical evaluations

Explanation. Attacks with similar distortion differ because they leave different amounts of the watermark signal in the image. For five post-hoc methods, Gaussian noise leaves 93 to 98% of the signal across its strengths, while diffusion regeneration leaves 32 to 42%: noise is added on top of the signal, while regeneration removes a large part of it. Detection follows the retained signal (rank correlation 0.63) more closely than the image distortion (minus 0.40). Detection can also remain after the message is lost: at 1% FPR a method is detected once its bit accuracy exceeds 0.57 to 0.65, while an uncoded message needs every bit to be correct.

For more robust watermarks. (a) Post-hoc methods need to combine two kinds of robustness that now exist in different methods: on real images, VINE keeps 92% detection (TPR@1%FPR) under regeneration but 34% under geometric attacks, and PixelSeal keeps 91% under geometric attacks but 15% under regeneration. (b) In-generation methods need geometric synchronization: all four fall below 50% detection on the same 10 of the 52 conditions, 8 of which contain cropping or rotation. (c) The message needs an error-correcting code: in 105 of the 247 post-hoc method-condition pairs with at least 90% detection, at most half of the messages are recovered exactly, and a BCH code raises the complete-message recovery of PixelSeal and VideoSeal under multiple perturbations (C3) from 0.553 to 0.96.

For new attacks. Diffusion regeneration followed by a mild crop and JPEG compression, which combines the weaknesses in (a) and (b), is the only condition that brings both families below 10% detection on the matched cohort.

In the paper: Section 5 (last paragraph), Section 5.2, Appendix F.10 (Tables 38 and 39).

W2: Insufficient validation of the ASI metric

We did both. At the same ASI (0.20, interpolated between measured strengths), mean detection of post-hoc methods on real images is 98.5% under Gaussian noise but 27.6% under diffusion regeneration (Table 32). With eight other image-quality metrics in place of LPIPS, the finding holds: every metric finds pairs of attack conditions in which the less distorted one is detected more than 50 percentage points less often (30 pairs under ASI, 3 to 62 under the others, Table 33).

In the paper: Section 4 C, Section 5.2, Appendix F.8.

Q1: Ablation with other degradation metrics

We added eight metrics: LPIPS-VGG, DISTS, SSIM, MS-SSIM, RMSE, NMI, CLIP distance, and the four per-image terms of the NeurIPS 2024 competition score (PSNR, SSIM, NMI and LPIPS, with their published weights). The main finding holds under each of them, and the method ranking changes little (Kendall tau 0.72 to 1.00). The other four terms (FID, CLIP-FID, aesthetics and artifacts) are left out because they are not distances between an attacked image and its own original.

In the paper: Appendix F.8.

W3: Limited practical relevance of QSR

QSR is reported next to complete-message recovery, not instead of it. Each question asks for one field of the message, so QSR counts how many fields are recovered. This matters when a watermark carries provenance fields as text, such as the creator or the generating model, a possible form of the machine-readable marking that the EU AI Act requires. Under the four image-sharing chains (resizing and JPEG compression), PixelSeal and VideoSeal recover 49% and 54% of the messages exactly, but their QSR is 0.78 and 0.83. Under the three removal attacks of the main suite it is below 0.01.

In the paper: Section 4 B, Section 5.5, Appendix F.7.

W4: Lack of qualitative analysis

We added both. Figure 5 shows PixelSeal images under two pairs of attacks. In each pair, the attack with the lower ASI removes the watermark and the attack with the higher ASI does not, which is a stricter test than similar ASI. The outcomes shown hold for 1,116 of the 1,193 images of the matched synthetic cohort. Table 37 shows recovered messages with their bit accuracy and QSR.

In the paper: Appendix F.9 (Figure 5, Table 37).

Q2: Design and selection of the 240 messages

The messages are built from fixed templates with randomly chosen words and fictional people, so no answer can be guessed from world knowledge, and each answer is a single word of the message.

To test whether QSR depends on the chosen messages, we recomputed it on 2,000 bootstrap resamples of the messages, using the scores already measured. The QSR of every method stayed within 0.04 of its reported value, and for 43 of the 45 pairs among the 10 message-carrying methods, the method with the higher QSR stayed the same in at least 95% of the resamples. The other two pairs, VINE against VideoSeal and against PixelSeal, differ in QSR by 0.010 and 0.025.

In the paper: Appendix D, Appendix F.5.

Reviewer_UuNLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SEALBench, a comprehensive benchmark for systematically evaluating the robustness of 15 image watermarking methods. It proposes the attack-strength index (ASI), which unifies all image-level watermarking attacks on a single damage axis, enabling more consistent comparisons. Additionally, the paper introduces Question Success Rate (QSR) as a metric to assess the extent to which embedded messages can be successfully recovered. Through extensive experiments, the paper provides a comprehensive evaluation of existing watermarking methods and reveals several key insights into their robustness.

Critical strengths

1. Novel Research Motivation: This paper addresses an important yet largely overlooked issue in the field of image watermarking: different types of attacks on image watermarks use parameters with different scales and value ranges, making it difficult to directly compare their attack strengths. The paper attempts to address this issue by proposing a new evaluation metric.

2. Large-Scale and Diverse Evaluation Data and Attack Settings: This paper establishes a large-scale benchmark for evaluating image watermarking methods, encompassing a substantial number of real, manipulated, and synthetic images. The dataset draws on multiple image generation models and covers diverse content categories ensuring broad data diversity. Furthermore, the evaluation includes 52 attack conditions spanning a wide range of attack types, providing a solid experimental foundation for systematically assessing the robustness of different watermarking methods.

Critical weaknesses

1.Limited Diversity of Distinct Attack Methods: Although the paper claims to cover 52 attack conditions, most of these are derived from different parameter settings of the same underlying attacks or sequential combinations of existing attack operations, rather than representing an equivalent number of distinct attack methods. As a benchmark, it should aim to provide more comprehensive coverage of diverse attack techniques by incorporating a broader range of dedicated watermark attack methods, rather than primarily relying on permutations and combinations of basic perturbations.


2.Concerns Regarding the Validity of the ASI Design: LPIPS primarily measures perceptual differences at the deep feature level, and its evaluation may be biased across different types of attacks, such as geometric transformations, local edits, and semantic modifications. Therefore, similar LPIPS scores do not necessarily imply that different attacks cause equivalent levels of actual image degradation. While using LPIPS to enable comparisons across attacks is reasonable to some extent, relying on it as the sole measure of attack strength introduces limitations that warrant further discussion.


3.Concerns Regarding the Validity of the QSR Metric: QSR may be influenced by the language model's own reasoning capabilities and prior knowledge. Specifically, the model may rely on linguistic priors, common sense, or educated guesses when recovering watermark messages, raising concerns about whether QSR truly measures watermark message recovery or merely reflects the language model's ability to infer or guess the intended content. Although the authors evaluate different readers, this does not fully rule out potential biases introduced by language model priors.

Questions for authors

In the field of invisible image watermarking attacks, attack methods are typically evaluated along two dimensions: Effectiveness, which measures the extent to which an attack disrupts watermark detection or message recovery, and Utility, which reflects the impact of the attack on image quality and is commonly assessed using metrics such as LPIPS, PSNR, and SSIM.

The central idea of this paper appears to be comparing the effectiveness of different attacks at similar levels of utility. However, the authors have not adequately justified why they prioritize comparing effectiveness at fixed utility levels, rather than also considering the complementary perspective of comparing utility at fixed effectiveness levels. This choice is particularly worth discussing given that different image utility metrics exhibit varying sensitivities to different types of image distortions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UuNL

Thank you for the careful review and for recognizing the motivation and scale of SEALBench. We address each point below, and all changes are in the revised PDF.

W1: Limited diversity of distinct attack methods

We agree that the suite needs more dedicated watermark-removal attacks, and we added two. SEALBench now evaluates five such attacks: VAE regeneration, diffusion regeneration and UnMarker in the main suite, and CtrlRegen and the adversarial embedding attack of WAVES, which we ran on all 15 methods and all three image categories and report separately from the 28 attack types (Appendix F.11, Table 40).

The two attacks have different effects. For post-hoc methods, CtrlRegen is stronger than every attack type in the suite: its mean AURSC is 0.53, 0.54 and 0.51 on real, manipulated and synthetic images, against 0.62, 0.61 and 0.57 for diffusion regeneration, the strongest of the 28 types, and almost no message is recovered (QSR at most 0.12). The adversarial embedding keeps mean TPR@1%FPR above 94% at every ε, but at ε = 8/255 it removes the messages of PixelSeal and VideoSeal on all three image categories (QSR falls from at least 0.94 to at most 0.03). Because of limited GPU time during the rebuttal, these two attacks are currently evaluated on 40 images per method for each image category (post-hoc) or generator (in-generation). We will run them on the complete set for the camera-ready version.

On the existing suite: the 52 conditions come from 28 attack types (14 single types and 14 fixed chains), and AURSC gives each type equal weight, so the settings of one type count once. The single types are not variants of one another: on the matched synthetic cohort, the Spearman correlation between the method rankings of two single types has a median of 0.32 over the 91 pairs, and the most damaging type differs between methods (diffusion regeneration for 5 of the 11 post-hoc methods, rotation for 3, VAE regeneration for 2, JPEG for 1). Averaging over the 14 single types alone changes the post-hoc ranking little (Kendall tau 0.86 to 0.96).

In the paper: Section 3.3, Appendix F.3, Appendix F.11 (Table 40).

W2: Validity of the ASI design

ASI keeps LPIPS as its default, and we now test whether the results depend on it: Section 4 C defines ASI as a protocol in which any metric that compares the attacked image with its original can replace LPIPS. We repeated the analysis with eight such measures, including DISTS, SSIM, RMSE (which orders images as PSNR does) and CLIP distance. Our main finding holds under every one: for post-hoc methods on real images, some attacks distort the image less than others but leave detection more than 50 percentage points lower (Table 33). The measures do differ on the attack classes you list: DISTS and CLIP distance rate cropping and rotation as milder than LPIPS does, CLIP distance rates the prompt-based edits as more distorting, and the pixel measures (SSIM, RMSE) agree least with LPIPS on the model-based attacks. The method ranking changes little (Kendall tau 0.72 to 1.00).

In the paper: Section 4 C, Section 5.2, Appendix F.8 (Tables 33 to 36).

W3: Validity of the QSR metric

Beyond comparing readers, we tested the priors directly with all five readers on the 840 questions of the message corpus (Table 26). With an empty message, every reader answers "unknown". With the answer word deleted or replaced by a random string, no reader answers more than 2 of the 840 correctly, and with a message that does not contain the answer, none. On attacked images, 98.5% (post-hoc) and 99.5% (in-generation) of the primary reader's correct answers come from answer words decoded without any error. Correct answers therefore come almost entirely from the decoded text. A reader-free score, which credits a question when its answer word is decoded exactly, orders the methods as QSR does within each message length (Table 27).

In the paper: Appendix F.5 (Tables 26 and 27).

Q: Fixed utility vs fixed effectiveness

We now report both views. Table 14 gives the fixed-effectiveness view, the ASI at which mean TPR@1%FPR falls to 50%. For post-hoc methods, only 4 of the 12 attack types measured at several strengths reach 50% within the measured range (diffusion and VAE regeneration, rotation and cropping), so this view leaves most attacks without a value. Fixed distortion is defined for every attack and method, which is why it is our main view. The two views also order the attack types similarly: AURSC and the ASI at which detection falls to 90% have a Spearman correlation of 0.87 to 0.92. Under every other distortion measure the same four types reach 50% (Table 35).

In the paper: Section 4 D, Appendix F.3 (Table 14), Appendix F.8 (Table 35).

Thank you for the detailed response and additional experiments. My concerns have been partially addressed, but I maintain my original rating.

My main concern remains the design of both evaluation metrics. If you defines ASI as a protocol in which any metric that compares the attacked image with its original can replace LPIPS, it is unclear whether ASI represents a well-defined measure of attack strength or merely an evaluation protocol, since different metrics capture different aspects of image degradation. For example, LPIPS measures perceptual feature differences, RMSE measures pixel-level differences, and CLIP distance captures differences in image representations.

Moreover, although both fixed-utility and fixed-effectiveness evaluations are theoretically valid, I still consider fixed-effectiveness evaluation more appropriate. Image quality is inherently multidimensional, making it difficult to establish truly equivalent levels of image utility across different images.

AuthorsOpenReview ↗

Reply to Reviewer UuNL

Thank you for the follow-up. We address your two points below.

  1. ASI is a protocol with LPIPS as the default metric

ASI is a protocol with two parts. The value is a well-defined measure, because the ASI of an attacked image is its LPIPS-AlexNet distance from its own unattacked image (Section 4 C). The procedure is how the value is used, so all attacks are placed on this axis and robustness is reported over a fixed range of it. LPIPS-AlexNet is the default metric, and in every table ASI denotes this value. Results under another metric carry that metric's name (Tables 33 to 36). LPIPS is the default because it agrees with human judgments of image similarity better than PSNR and SSIM (Zhang et al., CVPR 2018). Laidlaw et al. (ICLR 2021) likewise use an LPIPS bound as one threat model that covers several types of attack.

The eight other metrics are a sensitivity test. They capture different aspects of degradation, as you note, and the finding holds under each of them. Taking your three examples, for post-hoc methods on real images the number of pairs of attack conditions in which the less distorted condition is detected more than 50 percentage points less often is 30 under ASI (LPIPS-AlexNet), 50 under RMSE and 62 under CLIP distance, and between 3 and 62 over all eight metrics (Table 33). The metric changes the size of the effect but not its presence. The method ranking also changes little under all eight metrics (Kendall tau 0.72 to 1.00).

  1. Fixed effectiveness

Fixed effectiveness reports the distortion at which detection falls to a chosen level. We report it in Table 14 as the ASI at which mean TPR@1%FPR falls to 50%, and in Table 35 under the eight other metrics. We keep fixed distortion as the main view for four reasons.

(a) Fixed effectiveness also depends on the image-quality metric. The quantity it reports is a distortion value, so it must be measured with LPIPS or with another metric, and the order of the attacks changes with the metric. In Table 35, rotation reaches 50% at lower distortion than both regeneration attacks under DISTS and CLIP distance, and at higher distortion under ASI and the other six metrics.

(b) Fixed effectiveness at 50% has no value for most attacks within the measured settings. Table 14 covers the 12 attack types measured at several strengths. For post-hoc methods, 4 of them reach 50%, namely diffusion regeneration at ASI 0.11 to 0.13, VAE regeneration at 0.20 to 0.23, rotation at 0.23 to 0.25 and crop at 0.47 to 0.49. The other 8 do not reach 50% at any measured setting, and 7 of them stay above 87%. For in-generation methods, only crop and rotation reach 50%. At this level, the remaining attacks cannot be ranked among themselves. AURSC gives a value for every attack type and method.

(c) Fixed distortion does not require exactly matched images. ASI is computed for each attacked image against its own original, and AURSC integrates detection over ASI from 0 to 0.2 for each attack type. Our first finding is stated as an ordering, namely that an attack that distorts the image less can still reduce detection much more. On real images, VAE regeneration has a lower mean ASI than Gaussian noise (0.278 against 0.386) but leaves 34.9% detection against 89.5%, and seven of the eight other metrics, all except CLIP distance, also rate VAE regeneration as less distorting.

(d) The two views order the attack types similarly. Over the 12 attack types, the Spearman correlation between AURSC and the ASI at which detection falls to 90% is 0.87 to 0.92 for post-hoc methods, with types that stay above 90% placed at the robust end.

Both views have precedent. Adversarial robustness benchmarks report accuracy at a fixed perturbation budget (Madry et al., ICLR 2018, Croce et al., RobustBench, NeurIPS 2021), and minimum-distortion evaluation is used with attacks that succeed on every input (Carlini and Wagner, IEEE S&P 2017). In the final version, the main text will report the fixed-effectiveness results of Table 14 beside the AURSC results, with fixed distortion as the primary view.

Reviewer_v33wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SEALBENCH evaluates 15 image-watermarking methods under 52 attack conditions, covering real, manipulated, and synthetic images. Its main contributions are a matched cross-family evaluation protocol, LPIPS-based attack-strength reporting, and Question Success Rate (QSR), which measures whether a frozen language model can answer questions from decoded watermark text. The experiments examine detection and message recovery across attacks and generators, with a five-reader sensitivity analysis for QSR.

Critical strengths

1.Matching generators, prompts, and seeds improves cross-family comparability. The broad method coverage and evaluation of attacked negatives make the benchmark potentially valuable for subsequent watermarking research. 2.QSR complements bit accuracy and exact-message recovery by assessing whether partially recovered text remains useful. The five-reader analysis provides meaningful evidence about the metric’s sensitivity.

Critical weaknesses

1.ASI applies an existing perceptual distance, while broader coverage and matched evaluation are primarily protocol contributions. QSR is the most distinctive addition, but the paper should establish which practically meaningful conclusions or method-selection decisions it enables beyond exact-message recovery. 2.In the evaluated 256-bit setting, 140 bits encode at most 28 characters, so the task primarily measures short-message fact recovery. The practical value of partially recovering natural-language text, rather than recovering an identifier, needs clarification. Image hiding [1] and generative steganography [2–4] provide relevant context on richer payloads, encoding efficiency, and covertness. Positioning QSR against these objectives would clarify its contribution to the capacity–covertness–robustness trade-off.

[1] Baluja. Hiding Images in Plain Sight: Deep Steganography. NeurIPS, 2017. [2] Schroeder de Witt et al. Perfectly Secure Steganography Using Minimum Entropy Coupling. ICLR, 2023. [3] Ding et al. Discop: Provably Secure Steganography in Practice Based on “Distribution Copies”. IEEE S&P, 2023. [4] Wang et al. SparSamp: Efficient Provably Secure Steganography Based on Sparse Sampling. USENIX Security, 2025.

Questions for authors

Could the authors provide a concrete example, ideally from the existing evaluation, where QSR supports a practically meaningful conclusion or method-selection decision that exact-message recovery alone would miss? What downstream application makes partial recovery useful for the short payloads considered here?

1 comment
AuthorsOpenReview ↗

Response to Reviewer v33w

Thank you for your review and for the suggestions on how QSR is used and on its relation to steganography. We answer each point below, and all changes are in the revised PDF.

W1: ASI applies an existing perceptual distance

We agree, and we now describe ASI as a protocol for comparing attacks at similar distortion, not as a new metric. Our finding with ASI, that an attack which distorts the image less can still reduce detection much more, does not depend on the distance used: under each of eight other measures (such as DISTS, SSIM and CLIP distance), some attacks distort the image less than others but leave detection of post-hoc methods on real images more than 50 percentage points lower, and the method ranking changes little (Tables 33 and 34).

In the paper: abstract, Section 1, Section 4 C, Appendix F.8.

W1 and Q1: A method-selection decision that exact-message recovery would miss

We added one. PRC, PixelSeal and VideoSeal carry messages of the same length (140 bits, 28 characters), and all three have QSR of about 0.98 without attack. On the matched synthetic cohort, over all attacks, exact (complete-message) recovery ranks PRC first (0.54, against 0.48 and 0.46), but QSR ranks PixelSeal and VideoSeal above PRC (0.63 and 0.61, against 0.53). This matters when a method is chosen before an error-correcting code: PixelSeal and VideoSeal often fail by a few wrong characters, which a code repairs, while PRC returns no message when it fails. When the recorded bit errors are applied to BCH-coded messages, exact recovery becomes 0.72 and 0.70 for PixelSeal and VideoSeal, against 0.54 for PRC, which is the order that QSR gives. The reversal holds under all five readers, when any one of the 28 attack types is left out, and in all 2,000 message resamples.

In the paper: Section 4 B, Section 5.5, Appendix F.7 (Table 30).

W2: Positioning QSR against steganography

We added a paragraph on steganography to Section 2 that cites [1] to [4]. These works aim at covertness (provable security) and capacity (embedding rate), and robustness of the hidden content to later processing is not among their objectives. SEALBench measures this robustness at a fixed payload, the largest message class that fits each method (at most 11 or 28 characters, Appendix D.1), and QSR counts how many fields of the message survive. Your point on short payloads and identifiers is answered under Q2.

In the paper: Section 2.

Q2: Downstream application of partial recovery

The EU AI Act requires machine-readable marking of AI-generated content, and a watermark can carry such marking as a few provenance fields in text, such as the creator and the generating model. Each field is useful on its own, so recovering most of the fields has value even when the full message is not exact. An identifier, in contrast, is useful only when it is recovered exactly, so we report QSR next to exact recovery. Our sentences are an analogue of such a payload, and each question asks for one field. For example, under the four chains that apply the resizing and JPEG compression of image sharing (messenger, social, reshare and meme), PixelSeal and VideoSeal recover 49% and 54% of the messages exactly, but their QSR is 0.78 and 0.83: most fields survive, which exact recovery alone does not show. Under the three watermark-removal attacks of the main suite, their QSR falls below 0.01.

In the paper: Section 4 B, Appendix F.7.

Reviewer_V59wOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SEALBENCH, a benchmark that evaluates invisible image watermarking. It contains 4 in-generation and 11 post-hoc methods on real, manipulated, and synthetic images under 52 attack conditions across four categories: single perturbations, model-based editing and removal, multiple perturbations, and removal followed by multiple perturbations. The main 3 contributions are as follows: (1) SEALBENCH provides a synthetic cohort where both watermark families are compared on images from the same generator, prompt, and seed; (2) attack-Strength Index (ASI) is proposed to compare attacks on a single damage axis; (3) question Success Rate (QSR) is introduced, which uses a frozen LLM answer questions on decoded text to evaluate embedded message recovery. The paper finds attacks with similar mean ASI can affect detection very differently, such as (1) Gaussian noise and diffusion regeneration; (2) when evaluated under the matched synthetic cohort, neither post-hoc nor in-generation methods performed better; (3) high bit accuracy does not ensure message recovery.

Critical strengths

The paper makes cross-family comparison more reasonable by addressing a confound in literature. It provides a controlled setting: combine post-hoc and in-generation methods on the same images, prompts, seeds, and message corpus, with threshold calibration with respect to the unwatermarked and attacked-unwatermarked negative examples. The authors also make significant efforts in providing as much transparency and scale as possible. For example, the benchmark contains millions of attacked images, and the authors release the specific calibration thresholds, image states, and how precisely the attacks are conducted.

Critical weaknesses

The claim that “high bit decode accuracy doesn’t imply watermark message recovery” depends on complete-message recovery and QSR, which both depends on not only the watermark but also the payload encoding. Consider when real systems employ error correction (e.g. BCH). In Appendix D.1, the message is packed as raw 5-bit characters without error correction. Error correction could be applied to fix bit flips before decoding into text, which means the gap between bit accuracy and QSR (Figure 2) could’ve been influenced by this design. As additionally noted by authors in Appendix F.5, QSR depends on the reader, particularly on short messages, so QSR-based ranking results should be reviewed with caution.

Questions for authors

Does the bit-accuracy vs QSR gap pattern continue even after employing error correction code protocols pre-embedding? If not, the claim in the abstract could become weaker as it may describe payload encoding as much as the watermark.

1 comment
AuthorsOpenReview ↗

Response to Reviewer V59w

Thank you for your positive review and for the two questions on error correction and on the reader. Both led to new analyses, which are now in the revised PDF.

W1 and Q: Does the gap remain with error correction?

We tested this. We applied the bit errors recorded for every attacked image to the same message protected by an error-correcting code (BCH), without embedding or attacking any image again. The code uses the positions that hold zero padding in the uncoded payload, so it takes no capacity from the message. The code makes the gap smaller but does not remove it, because it repairs only a limited number of wrong bits: for a 28-character message, the BCH codeword is 248 bits long and corrects at most 14 of them. With BCH, the complete message is recovered for 97% of attacked post-hoc images with bit accuracy between 0.95 and 0.99, but for only 21% of those between 0.90 and 0.95. In Figure 2 (matched synthetic cohort, one point per post-hoc method and attack condition), 212 points have bit accuracy of at least 0.9. With BCH, the number of these with QSR of at most 0.5 falls from 42 to 25 (Table 31). The abstract now states that the finding is for an uncoded payload and that SEALBench measures recovery both without and with a code.

In the paper: abstract, Section 5.5, Appendix D.1, Appendix F.7 (Tables 30 and 31).

W2: QSR depends on the reader

Appendix F.5 runs the complete reader pass with five readers. Averaged over methods, mean attacked QSR differs by at most 0.08 between readers, and condition-level QSR is strongly correlated between every pair of readers (Pearson r of at least 0.985). The three lowest post-hoc methods and the order of the in-generation methods are the same under all five readers. Only the order of the four highest post-hoc methods changes, mainly because of VINE and OmniGuard, which carry short messages (at most 11 characters), on which the readers differ most. We therefore report QSR next to complete-message recovery. We also added a reader-free score, which credits a question when its answer word is decoded exactly. It gives the same method order as QSR within each message length (Table 27).

In the paper: Section 5.5, Appendix F.5 (Tables 22 to 27).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We thank all reviewers. The list below gives each change in the revised PDF and the reviewer points it answers. The main text remains within 10 pages.

Main text

  1. Abstract and Section 1: ASI is described as a protocol for comparing attacks at similar distortion, and the abstract states that the bit-accuracy finding is for an uncoded payload and that we measure message recovery both without and with an error-correcting code. In Section 1, the first finding uses an example in which the less distorting attack (VAE regeneration, mean ASI 0.278) reduces detection more than Gaussian noise (0.386) (UuNL W2, QeTC W2, v33w W1, V59w W1).
  2. Section 2: a new paragraph on steganography states that these methods aim at security and embedding rate, whereas SEALBench measures robustness at a fixed payload (v33w W2).
  3. Section 3.3: the attack suite is described as 14 single attack types and 14 fixed chains, with a pointer to two further watermark-removal attacks in Appendix F.11 (UuNL W1).
  4. Section 4 B: an example in which the message with the higher bit accuracy keeps fewer facts, and three uses of QSR: next to exact recovery for identifiers, for counting provenance fields, and for choosing a watermark before an error-correcting code (QeTC W3, v33w W1/Q1/Q2).
  5. Section 4 C: the analysis is repeated with eight other image-quality metrics, and the main finding holds under every metric (UuNL W2, QeTC W2/Q1, v33w W1).
  6. Section 4 D: why robustness is compared at fixed distortion, and where results at fixed effectiveness are reported (UuNL Q).
  7. Section 5.2: the same example on real images (UuNL W2, QeTC W2), and how much of the watermark signal these two attacks leave in the image for five post-hoc methods (QeTC W1).
  8. Section 5.5: the bit-accuracy finding is stated for a payload without an error-correcting code, and a new paragraph shows how QSR changes a choice between methods (V59w W1/Q, QeTC W3, v33w W1/Q1).
  9. Section 5, last paragraph: what the results imply for watermark design and attacks (QeTC W1).

Appendix

  1. D: the messages are generated from fixed templates (QeTC Q2). D.1: the payload has no error-correcting code, and Appendix F.7 tests codes (V59w W1).
  2. F.3: method ranking over the 14 single attack types only, how differently the attack types rank the methods (UuNL W1), and results at fixed effectiveness (Table 14, UuNL Q).
  3. F.5: reader controls and a score that uses no reader (Tables 26 and 27, UuNL W3, QeTC Q2, V59w W2), and a resampling of the messages (QeTC Q2).
  4. F.7 (new): message recovery with Hamming and BCH codes, and recovery of single message fields under image sharing (Tables 30 and 31, V59w W1/Q, QeTC W3, v33w W1/Q1/Q2).
  5. F.8 (new): ASI under eight other image-quality metrics (Tables 32 to 36, UuNL W2/Q, QeTC W2/Q1, v33w W1).
  6. F.9 (new): case studies: images in which the attack with the lower ASI removes the watermark (Figure 5), and recovered messages with their bit accuracy and QSR (Table 37, QeTC W3/W4).
  7. F.10 (new): the measurements behind five findings and behind the implications in Section 5 (Tables 38 and 39, QeTC W1).
  8. F.11 (new): two further watermark-removal attacks, CtrlRegen and the WAVES adversarial embedding, on all 15 methods (Table 40, UuNL W1).
Area_Chair_2PGXOpenReview ↗

Please, start a conversation

Dear reviewers,

The discussion period is about to end and the authors have responded to your reviews. This is a friendly reminder to check these responses and respond if necessary.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.