acceptodds
Withdrawn from ICLR 2027

The Search for Statistical Indistinguishability Between Real and Fake Images

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Images produced by generative models appear strikingly similar to real ones. In this work, we study whether perfection has been reached by generated content in terms of matching the properties of real data. We train deep neural networks to separate real data from fake and observe their test accuracies. Only a chance performance has any scope of proving statistical indistinguishability. However, since generative models are known to sometimes introduce small errors, e.g., low level checkerboard artifacts, we create progressive abstractions of the original real/fake images to potentially bring them to a space where those artifacts get wiped away. These include operations such as blurring, downsizing, cropping, converting RGB images into edge/semantic maps, etc. To our surprise, almost none of the abstractions are able to make fake data indistinguishable from real. The only cases when neural network based classifiers fail to separate the two categories is when the data is abstracted to such an extent that the original content seems mostly lost to the human eye, e.g., downsizing a RGB image of human face to resolution. We observe this trend across four different datasets of varying complexity – MNIST, CelebA, FFHQ and GenImage – and across various types of generative models – GANs, VAEs, diffusion models. This shows that the nature of fake artifacts extends all the way from low level to very high level in images. The simple implication is that fake images are much more different from real than we are used to think.

Primary area:
generative models
Keywords:
generative models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_gfnrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an investigation of the limits of the detection of generated images. It uses "image abstractions", which are basically simple image augmentations like blurring, noising or down sampling, on which it trains detector networks to show that generated images are still detectable even under strong "abstractions".

Critical strengths

The paper addresses the timely and important question whether the distributions of generated images already match the training data or further improvement in generation methods is needed.

Critical weaknesses

The paper suffers from several severe problems, including readability, completeness of related work, ignoring SOTA generators, over-claiming, very low novelty and a questionable experiment design.

The title suggest an in-depth statistical analysis of complex image distributions - However, what the reader gets is simple off-the-shelf binary classification of images under augmentations.

The paper is hard to read, not because it is so complicated - rather the opposite: it stretches trivialities in lengthy passages that hardly add new information for the reader.

The related work section is rather outdated, referring to mostly older works. This is also the case for the selected generator models which are quite old: the newest is from 2023. This results in low-resolution experiments only.

The novelty of the insights provided is very low, as the authors write them self: it is basically standard procedure to use (the same) augmentations in benchmarks to test the robustness. The claimed novelty come from questionable setup: trying to overcome the separation of OOD data by the application of (mostly) linear augmentations is doomed to fail in any case. This is what the experiments actually show.

Questions for authors

None. I really would be interested in an in-depth statistical analysis of image-distributions from modern generative models, showing where and why they fail(?) to match real data. However, this would require a completely different approach..

Reviewer_tfykOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether statistical differences between real and generated images persist across different image representations. The authors apply transformations such as blurring, downsampling, cropping, edge extraction, semantic segmentation, and landmark extraction, and train separate binary classifiers from scratch for each setting. Experiments on MNIST, CelebA, FFHQ, and GenImage show that real and generated images remain distinguishable under many transformations. The paper interprets these results as evidence that generative errors extend beyond isolated low-level artifacts. Its contribution is primarily an empirical investigation rather than a new detection method.

Critical strengths
  1. Investigating which aspects of generated images remain statistically distinguishable from real images is useful for understanding generative model fidelity and forensic signals.

  2. Training a new classifier for each representation directly examines whether discriminative information remains after transformation. This protocol is appropriate for the question being studied.

  3. The experiments cover several datasets, generator families, classifier architectures, and transformation strengths. The appendix also includes useful controls, including real-versus-real classification, digit balancing, and format/resolution matching.

Critical weaknesses
  1. The empirical advance needs to be established more clearly. The classifier-based test and the image transformations are established tools, so the contribution depends on the novelty and explanatory value of the findings. The paper’s own related work discusses detection using suppressed high-frequency information, learned representations, and geometric properties. It remains unclear which previous understanding is substantially revised by the present results. Most experimental subsections repeat the same classification protocol on different datasets, providing breadth but limited analysis of competing explanations. A clearer account of the new insight, supported by targeted controls, would strengthen the contribution.

  2. There is an inferential gap between the experimental evidence and the central conclusion. The experiments directly establish that real and generated images remain distinguishable under the evaluated datasets and transformations. However, the further interpretation that generative errors span low-to-high-level image properties is not sufficiently supported. Different transformations may preserve overlapping discriminative cues, while differences in segmentation masks or landmarks may partly reflect systematic responses of the extractors rather than high-level modeling errors of the generators. As an empirical study, the paper needs targeted controls to clarify the sources of these differences and how they support the conclusion that generative errors are broadly distributed.

  3. Generation-related differences are not sufficiently separated from data-source and preprocessing differences. The existing controls are useful but do not fully resolve this issue. For example, the GenImage real images originate as JPEG photographs with varying aspect ratios, whereas generated images originate as square PNGs at different resolutions. Converting both classes to PNG and matching their final resolution does not remove their compression histories or differences in cropping and resampling. Differences in content and appearance distributions may also remain. The experiments therefore establish separability of the evaluated datasets more directly than they establish that the separating signals arise specifically from generative errors.

  4. The statistical criterion for indistinguishability is insufficiently specified. The paper acknowledges that chance-level classifier performance cannot prove distributional equality. Nevertheless, descriptions such as “50–55%” being approximately chance require calibration, particularly with large test sets. Repeated runs, uncertainty estimates, and a statistically justified decision criterion are needed for these cases. Reporting the best classifier also calls for a corresponding null calibration when interpreting small deviations from chance. Furthermore, the classifier test concerns marginal distribution differences; it does not establish independence between the real and generated sample sets. These issues should be distinguished more clearly in the formal discussion.

Minor Issues

Figure 1 reports 52.6% for the 1×1 crop, whereas Table 1 reports 54.8%; 52.6% corresponds to 1×1 downsampling in that table. The CelebA appendix also gives both 25 and 50 training epochs. Please resolve these inconsistencies and specify the landmark extractor and its input encoding sufficiently for reproduction.

Questions for authors
  1. What is the principal new empirical insight beyond existing observations of detectability using low-frequency, geometric, or learned representations?
  2. Can additional controls disentangle generation effects from compression history, resampling, and differences in content or appearance statistics?
  3. Can simple statistical baselines and alternative or manually verified structural representations clarify which cues survive each transformation?
  4. What criterion determines that accuracy is sufficiently close to chance? Can the authors provide repeated runs, uncertainty estimates, and calibration accounting for the classifier family?
Reviewer_UdEmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes an empirical analysis regarding statistical indistinguishability between real and fake images identified by the detector. Even though when generative artifacts are removed by performing image transformation like blurring, down sampling, the trained classifiers are still able to learn a discriminative signal, that separates real and fake data with high accuracy. The classifiers fail only when the image quality/content is significantly destroyed from a human-eye perspective. Hence, this work claims that distinguishable signal between real and fake domain expands from coarse to fine details.

Critical strengths

The problem studied in this work overlaps with an important research domain of synthetic image detection.

Critical weaknesses
  1. In this work, a new classifier is trained for each level of abstract transformation. The authors claim that when a classifier is trained from scratch on a new transformation like blurring, adding noise (which potentially destroys fine-grained visual cues), the classifier is still able to detect fake images. I believe the reason for training a new classifier for a different level of transformation is that cross-transformation performance (i.e., different transformation seen during test-time) in terms of detection accuracy is not good enough? From that angle, the novelty of this work is limited. It is trivial to realize that when a detector is trained from scratch with dataset including a new transformation, it would demonstrate high accuracy on test dataset with the same level of transformation, unless the images collapse in terms of content or quality. I would say - searching for a detector that is robust against new unseen transformation would be more interesting and novel. A lot of works has been done in terms of developing generalized detectors like [R1], [R2], that show good detection performance on unseen fake dataset.

  2. The paper claims that a distinctive signal exists even at low-level details between real and fake images. However, it is not discussed what type of signals such detectors rely on. For example, is it some distinctive signal in the frequency domain, spatial domain, discrepancy in intermediate embeddings of some foundation model or something else? A thorough analysis or at least some intuition would be beneficial to support the claim.

  3. As shown in the experimental figures, blurring, down sampling or cropping to such aggressive scale, e.g., 4x4 down sampling is particularly irrelevant in practical applications of synthetic image detection, where a deployed system wants to identify high-quality, high-resolution synthetic images, that may appear real in human-eye. It is highly unlikely that any detection system in research industries would consider images with such aggressive transformation, that fully collapses the image content. Demonstrating the detection success under such aggressive transformation seems little of interest to the relevant research community.

  4. Regarding detectors, authors show experiments with ResNet, ViT etc. It would be ideal to consider recent synthetic image detectors like [R1], [R3].

  5. The authors can further look into how real vs. fake images after different transformations behave in the feature space of large-scale pretrained models like in [R4]. This would potentially help assess whether detection methods leveraging foundation model's feature representation can generalize across various transformations discussed by the authors.

[R1] Yan, Z., Wang, J., Jin, P., Zhang, K., Liu, C., Chen, S., Yao, T., Ding, S., Wu, B., & Yuan, L. (2024). Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection. International Conference on Machine Learning.

[R2] Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023). UCF: Uncovering Common Features for Generalizable Deepfake Detection. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 22355-22366.

[R3] Wen, S., Ye, J., Feng, P., Kang, H., Wen, Z., Chen, Y., Wu, J., Wu, W., He, C., & Li, W. (2025). Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation. Neurips 2025.

[R4] Ojha, U., Li, Y., & Lee, Y.J. (2023). Towards Universal Fake Image Detectors that Generalize Across Generative Models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Questions for authors

Please see the weaknesses above.

Reviewer_QMkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether differences between real and generated images persist beyond low-level artifacts. The authors apply various image transformations and train new classifiers for each representation. Across four datasets and several generative models, real and generated images remain distinguishable under most transformations, approaching chance-level accuracy mainly when image content is severely degraded. The findings suggest that discrepancies persist across diverse representations, despite the visual realism of generated images.

Critical strengths
  1. I think training a new classifier for each representation is a good choice. A fixed detector may fail simply because it was not trained for transformed images. Retraining makes the experiments better suited to checking whether useful detection cues still remain.

  2. The experiments cover several datasets and different types of generative models, which gives the observations some breadth. I found the semantic-mask and landmark results particularly interesting, as they show that differences remain even after much of the original image information is removed.

Critical weaknesses

(1) One concern with the GenImage experiments is that the classifiers may be picking up differences in how the images were processed, rather than errors introduced by the generative models. Appendix B.4 states that the real images are originally JPEGs and the generated images are PNGs, with different native resolutions and aspect ratios. The authors do match the final resolution and file format and apply the same transformations to both classes. Still, saving a JPEG as a PNG does not undo its earlier compression, and matching the final image size does not necessarily remove differences caused by resizing. It is therefore worth checking whether these processing cues survive the abstractions and contribute to the reported accuracy.

The real-vs-real control does not quite address this concern. Both groups are randomly drawn from the same pool of real images, so compression history does not systematically distinguish one group from the other. A classifier could therefore perform at chance on this control while still relying on compression differences to separate real from generated images. The relevant paragraph in Appendix B.4 also describes the control without reporting numerical results. Additional controls would help clarify how much of the GenImage result reflects the generated content itself and how much may come from differences in processing history.

(2) A second concern is that the additional insight beyond existing evidence of detectable artifacts remains limited. Chai et al. (2020), for example, showed that generated images retain detectable local artifacts even after a generator is adjusted to evade an existing classifier. Their experiments differ from the abstraction-based analysis here, but they already establish the broader point that convincing visual quality does not imply the absence of detectable differences. The present paper extends this evidence across multiple representations, yet it is less clear what new understanding emerges from that extension. For example, the landmark results show that real and generated faces are separable, but do not identify which aspects of the extracted landmark configurations drive the difference. Understanding what causes these signals, rather than primarily documenting their persistence, would offer a more substantive insight and could help guide efforts to make generated images better match real data.

Questions for authors
  1. The numerical results of the real-vs-real control in Appendix B.4 should be reported. Additional evidence is needed to establish that differences in compression and resizing history do not substantially contribute to the GenImage results.

  2. The paper needs to clarify what specific insight goes beyond prior evidence of persistent detection cues, such as Chai et al. (2020). In particular, an analysis of which properties of the extracted landmark configurations drive the separation would help establish the significance of these results.

  3. The interpretation of accuracies around 50–55% as near chance needs stronger justification. Given the large test sets, some of these values may still be significantly above 50%. Confidence intervals or appropriate statistical tests are needed to distinguish numerical closeness to chance from a lack of statistically significant separation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.