acceptodds
Under review as a conference paper at ICLR 2027

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

Abstract

AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana Pro. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.

Primary area:
datasets and benchmarks
Keywords:
dataset, AI-generated image detection, synthetic image detection, text-to-image models, deepfake detection, robustness evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1ULVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose REALIS, a large-scale dataset for benchmarking AI-generated image detectors with particular focus on generalizability across unseen image generators and robustness to common image transformations and corruptions. The dataset is generated with several generators, including recent and commercial ones, and applies several filtering and sampling steps to mitigate biases and shortcuts.

Critical strengths
  • The authors put a considerable effort into the dataset design, aiming to mitigate every possible bias and shortcut in the images that might be exploited by the detectors. This increases the reliability of the evaluation results.

  • The dataset includes a broad set of image generators (also recent and commercial ones) and image transformations/corruptions, making its scope wide and relevant.

Critical weaknesses
  • The novelty with respect to previous work is not clearly discussed and seems limited. In particular, a quite relevant overlap with the referenced Gushchin et al. (2026) appears, and the paper's contribution seems incremental to this work, which is also missing from the Tab. 3 comparison. Moreover, the main objectives and findings (assessing generalization and robustness) have already been addressed and demonstrated in prior work.

  • Some of the filtering steps do not seem properly justified, and no evidence is provided on their usefulness (e.g., an ablation study). Furthermore, some of them might remove "legitimate" content that a detector might encounter in real-world scenarios (e.g., images with text, screenshots, collages, stock images, and watermarked images). Finally, the intensive use of CLIP in the filtering/sampling process may bias the results, as many detectors are built on top of the CLIP backbone.

Questions for authors
  • Could you please better position your work with respect to Gushchin et al. (2026)?
  • Do you apply any measure to align image resolution, aspect ratio, and compression quality (beyond the described bounds on resolution and ratio)?
  • How do you explain the very high test performance on detectors when trained on REALIS?
  • How were the hardest generators chosen in the Expert subset?
Reviewer_nvF8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REALIS, a large-scale dataset and benchmark for systematically studying the challenges of AI-generated image detection. To address the distribution discrepancies between real and generated images in existing AI image detection datasets in terms of semantic content, image quality, and generation-related characteristics, the authors propose a systematic data construction pipeline. In addition, the authors construct a more challenging subset, REALIS-Expert. This subset uses local density-ratio estimation in a low-dimensional feature space to select samples from regions where the distributions of real and generated images highly overlap, thereby reducing the possibility of detectors relying on simple semantic and visual features for classification. The paper also designs a post-processing evaluation protocol consisting of 35 image transformations at five severity levels to assess robustness. The experimental results show that existing detection methods exhibit significant performance differences across different generators and generally suffer substantial performance degradation under complex post-processing conditions. Overall, this work provides a relatively comprehensive evaluation framework that helps reveal the limitations of current AI-generated image detection methods in real-world applications.

Critical strengths

REALIS applies similar filtering criteria to real and generated images and derives generation prompts from image content, reducing the possibility of models exploiting semantic differences for classification. Meanwhile, the work considers dataset bias, cross-generator generalization, and post-processing robustness, with clearly defined research objectives.

Critical weaknesses

The paper contains inconsistencies in dataset statistics that need clarification. For example, Section 3.1.1 (Filtering Cascade) states that approximately 13 million candidate images were initially collected, of which 2.9 million were retained. However, Figure 2 indicates approximately 16 million web images initially, with about 2.22 million retained. Section 3.1.2 (Model Selection) states that 33 open-source generators and 9 proprietary generators were used, whereas Appendix B.1 mentions 12 proprietary models in its dataset comparison description.

Questions for authors
  1. The paper uses density ratios in a 12-dimensional feature space to measure sample difficulty. However, this feature space may not sufficiently capture the low-level synthesis fingerprints of AI-generated images. Furthermore, the paper does not adequately explain why a 12-dimensional feature space and the threshold (|\log \hat{r}(x)| \leq 0.5) were chosen. How sensitive are the results to these hyperparameters?
  2. The ablation experiments on hard sample selection show that density-ratio filtering alone only reduces the average AUC from 0.826 to 0.809, whereas selecting more challenging generators and applying quality filtering lead to larger performance drops. This raises the question of how much of the difficulty of REALIS-Expert actually comes from the proposed density-based selection algorithm, which requires further analysis.
Reviewer_Q1DnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces REALIS, a curated benchmark for AI-generated image detection under generator, content, and post-processing shifts. It contains approximately 1.43 million real and synthetic images spanning 42 text-to-image generators. Its construction combines image-derived prompting, filtering of both classes, and prompt-disjoint splits to reduce semantic leakage and dataset-specific shortcuts.

REALIS-Expert provides a more challenging evaluation subset through distribution matching, generator selection, and quality filtering. The benchmark also includes paired clean and processed images, with a robustness protocol covering 35 transformations at five severity levels. Experiments evaluate pretrained detectors, REALIS-tuned models, and zero-shot vision-language models. Performance frequently deteriorates after processing; the best REALIS-tuned detector reaches 0.752 ROC-AUC on the processed open-source component of REALIS-Expert.

Critical strengths
  1. A thoughtful benchmark design targeting common detection shortcuts. Image-derived prompts and prompt-disjoint splits are sensible ways to reduce content differences and prevent related real and generated examples from leaking across splits. Coverage of modern open and proprietary generators makes the resource relevant to contemporary detection research. Importantly, the authors acknowledge that alignment in the selected feature space does not establish the absence of low-level forensic shortcuts.

  2. A broad robustness evaluation with useful diagnostic analyses. The processing protocol extends beyond resizing and JPEG compression to neural compression, watermarking, repeated compression, and compound chains. Paired clean and processed images support direct comparisons, while per-generator results and separate false-positive and false-negative analyses reveal failures hidden by aggregate ROC-AUC. These analyses make the benchmark useful for diagnosing detector behavior rather than simply ranking methods.

Critical weaknesses
  1. The Expert evaluation and operating-point analyses do not fully isolate their intended conclusions. Expert differs from the regular Test split in generator composition, quality filtering, and density-based selection. The paper reports sequential construction results, which are informative, but these do not fully disentangle the factors under matched conditions. Lower Expert performance therefore cannot be attributed specifically to improved class alignment. Additionally, comparing detectors at a default threshold of 0.5 is difficult to interpret without comparable score calibration. Validation-selected thresholds and recall at fixed false-positive rates would better support conclusions about operating-point reliability.

  2. Release documentation and reproducibility remain insufficiently clear. The stated 1.43-million-image curated dataset differs substantially from the counts in the comparison table, which reports 2.2 million real and 8.8 million synthetic images. These may refer to different construction stages, but the relationship should be explicit and consistent. Only a partial release is available during review, limiting assessment of the complete resource. Clearer documentation of release contents, redistribution permissions, privacy safeguards, and treatment of identifiable individuals or sensitive content is important for evaluating a benchmark built from web-sourced images.

Questions for authors
  1. Dataset composition and release: Can the authors provide a consistent accounting of the source pools, filtering stages, and final splits, reconciling the 1.43-million-image dataset with Table 3? Which images, metadata, prompts, and generator outputs are available during review, and which will be released subsequently?

  2. Expert construction: Beyond the sequential construction results, can the authors provide matched comparisons that vary density-based selection, quality filtering, and generator composition separately? How much of Expert’s additional difficulty remains when generator composition is held fixed?

  3. Operating-point evaluation: How do detector comparisons change with thresholds selected on validation data or with recall measured at fixed false-positive rates? Please also clarify whether any calibration is applied before using the common 0.5 threshold.

  4. Missing results: Why are off-the-shelf results on clean Test unavailable? Can these be completed, or can the authors explain the limitation preventing their inclusion?

  5. Responsible release: What redistribution permissions and privacy safeguards apply to the real-image sources and proprietary-generator outputs? How are identifiable individuals, sensitive content, and removal requests handled?

A comparison with transformations observed in actual image-sharing pipelines would strengthen the external relevance of the robustness protocol. Until then, conclusions should remain tied to the proposed distribution of transformation chains and severities, rather than being interpreted as representative of all real-world sharing conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.