- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose REALIS, a large-scale dataset for benchmarking AI-generated image detectors with particular focus on generalizability across unseen image generators and robustness to common image transformations and corruptions. The dataset is generated with several generators, including recent and commercial ones, and applies several filtering and sampling steps to mitigate biases and shortcuts.
Critical strengths
-
The authors put a considerable effort into the dataset design, aiming to mitigate every possible bias and shortcut in the images that might be exploited by the detectors. This increases the reliability of the evaluation results.
-
The dataset includes a broad set of image generators (also recent and commercial ones) and image transformations/corruptions, making its scope wide and relevant.
Critical weaknesses
-
The novelty with respect to previous work is not clearly discussed and seems limited. In particular, a quite relevant overlap with the referenced Gushchin et al. (2026) appears, and the paper's contribution seems incremental to this work, which is also missing from the Tab. 3 comparison. Moreover, the main objectives and findings (assessing generalization and robustness) have already been addressed and demonstrated in prior work.
-
Some of the filtering steps do not seem properly justified, and no evidence is provided on their usefulness (e.g., an ablation study). Furthermore, some of them might remove "legitimate" content that a detector might encounter in real-world scenarios (e.g., images with text, screenshots, collages, stock images, and watermarked images). Finally, the intensive use of CLIP in the filtering/sampling process may bias the results, as many detectors are built on top of the CLIP backbone.
Questions for authors
- Could you please better position your work with respect to Gushchin et al. (2026)?
- Do you apply any measure to align image resolution, aspect ratio, and compression quality (beyond the described bounds on resolution and ratio)?
- How do you explain the very high test performance on detectors when trained on REALIS?
- How were the hardest generators chosen in the Expert subset?