SEALBench: A Comprehensive Benchmark for Image Watermark Detection, Robustness, and Recovery

As concerns over the misuse of Generative AI models continue to grow, watermarking has emerged as a promising solution, embedding a detectable signal into generated or existing images. Existing image watermarking benchmarks have three limitations: 1) inconsistency across data for comparing in-generation and post-hoc watermarking methods, 2) uneven attack settings for image/watermark distortion, and 3) bit accuracy that does not measure whether the decoded message retains its meaning. To this end, we present SEALBENCH, an image watermarking benchmark that evaluates 4 in-generation and 11 post-hoc watermarking methods on 25,864 real, 15,000 manipulated, and 12,665 synthetic images. We further evaluate image-level watermarking attacks in four attack categories, from single image transformations to editing or watermark removal followed by additional processing, and propose the attack-strength index (ASI) to unify all image-level watermarking attacks on a single damage axis. We evaluate and propose Question Success Rate (QSR), i.e. embedded message recovery, by testing whether a frozen language model can accurately answer questions using the extracted text. Our evaluation of State-of-the-art methods on SEALBENCH reveal interesting findings: a) both in-generation and post-hoc methods remain vulnerable to attacks, although their robustness varies significantly across attack types, b) attacks that produce similar image distortion can have different effects on watermark detection and c) High bit accuracy does not necessarily imply complete message recovery or preservation of the message’s meaning. We believe SEALBENCH will support more consistent evaluation of image watermark robustness and encourage research on methods that preserve both watermark detection and message recovery under different attack settings.