acceptodds
Under review as a conference paper at ICLR 2027

Real Images, Fake Verdicts: When Detectors Fail on Fake-Looking Photos

Abstract

Synthetic-image detectors are evaluated almost exclusively on benchmarks whose real images are typical photographs. We study the opposite tail: genuine photographs that look fake to people — extreme lighting, improbable geometry, coincidences, unusual post-processing. We introduce FLORIDA, 795 such curated real photographs, and evaluate ten detectors across six families after a shared control-benchmark sanity check. Seven pass; six nevertheless flag FLORIDA as fake at 1.8×–33× their control false-positive rate (all p < 10−4), up to 80% for the strongest detector. A provenance-matched control shows web provenance alone elevates FPR for five of seven detectors, while a genuine content effect survives matching for four of seven (up to 5.5×). The failures are also nearly independent across detectors (median φ = 0.04): 95.5% of FLORIDA fools at least one detector, yet none fools all seven. One deployed detector shows no significant elevation, and retraining a linear probe with 396 fake-looking reals as hard negatives cuts FPR from 55.9% to 1.0%: robustness is attainable and cheap. A nine-way taxonomy of why images look fake finds no safe category (union FPR 91–98%), with each detector sensitive to a different one; fake-looking reals are more atypical in frequency space than the fakes themselves. “Fake-looking” is not one shared cue detectors accidentally learn, but a broad region of real-image space where each detector fails differently — with direct consequences for deploying detectors as evidence of authenticity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fake-looking images, real photographs, AI image detection, false positives, detector robustness, synthetic image detection, FLORIDA dataset, provenance bias, hard negatives, authenticity verification, detector failures, image forensics, out-of-distribution images, frequency artifacts, adversarial robustness, computer vision, fake image detectors, benchmark evaluation, visual anomalies, image authenticity, multimodal AI, detector retraining, forensic AI, image classification, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H7c2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper suggests that existing detectors are evaluated in settings where the real images are common everyday natural images. This overlooks a critical set of real images, ones that do not look generic, but are not AI-generated. The paper curates a benchmark of 795 such images and tests this very hypothesis. With a control group of natural looking real images, the authors measure the elevation in false positive rate (real classified as fake) to get a sense of this. The major takeaways are (i) Most detectors conflate implausible looking images with fake images, (ii) Artifacts in post-processing also happen to contribute to this. (iii) There are not a lot of images that are commonly misclassified by detectors (iv) This issue is simple to overcome via training. The authors also conduct other experiments to study spectral properties, impact of "salient" regions of the image etc.

Critical strengths
  1. The overall research question and finding is interesting. It is also one of practical importance, especially in domains such as art where there are real images which do not look like everyday scenes.
  2. The main experiment is rigorous with sufficient controls such as a reference COCO set (also with Matching), adding to the validity of the findings.
Critical weaknesses
  1. The paper writing could be improved I feel. I find the message very interesting, but the writing style made it extremely difficult for me to follow here. I will add specifics below, a) Abstract has too much information which makes it hard for the first time reader: for instance the control checks (which reduce 10 to 7 is something done for empirical rigor, which is good, but also served confusing for me to grasp that early on into the paper, also the control benchmark sanity was not clear to me so early on into the paper, the takeaway should have just been mentioned as multiple detectors falsely classify such real images as fake. I would suggest avoiding so many readings in the abstract in general, especially given that the paper is not about those particular numbers, but the broader trend). b) I felt elevation was a somewhat unnatural word to use in lines 22 and 88 (although it made sense after reading the intro) c) Line 155: What does interpretable mean here? To me it feels like the role of the control is to ensure that the detector is a reliable one (since it is not informative to test a poor quality detector) d) Section 4.4 directly starts with results, I feel it would be better to start with 1 or 2 sentences motivating what the experiment is trying to do and why these metrics are the right fit.
  2. The paper studies a nice topic, but I would like it if it would provide more predictive power. For instance, it suggests that there is no common type of abnormal real image where all detectors fail (sec 4.4). It also seems to not actually use information from the training data, algorithm, architecture, initialization etc to link the way in which these models fail to provide further insights. The paper in its abstract talks about detectors ranging from 6 families, but the regularities within those families are not used in the analysis. Natural questions that arise (i) what makes Haywoodsloan special here, (ii) why do CLIPDet (Cozzolino et al) and UFD (Ojha et al) behave differently when they have roughly similar training recipes.
  3. Section 4.9: It would be helpful if the authors can clarify how this is a true test of generalization, for instance, the 396 training images in this case might have significant overlap with the test images.
Questions for authors
  1. [1] argues that a major issue with fake-image detectors is their tendency to learn features associated with the real-image distribution, even though these associations are almost-always spurious. For instance, a detector trained on relatively low-quality GAN images might learn to associate high image quality with real images. Such associations can also indirectly influence what the detector treats as evidence for fake images: if natural scenes are strongly associated with real images, their absence might contribute to a fake prediction. I wonder whether the authors see a connection between this observation and the failures reported on FLORIDA. The authors also release a Corvi +, which claims to have less reliance of real features, it would be helpful to see what happens if this model is evaluated on the FLORIDA dataset.
  2. Sec 4.3 suggests that for Corvi, the entire discrepancy is an artifact of post-processing. Which intuitively makes sense, since Corvi is trained on 96x96 crops of the images, implying that it is less likely to pick up on global, semantic abnormalities, I would like to know the authors thoughts on this, as well as analysis of similar methods and their failure modes.
  3. Linked to above, but the paper seems to suggest that the simple way of overcoming this issue is with additional training data, but this assumes a certain awareness of the kind of images seen at test time. It would be nice to have ways to build generalizable detectors which do not assume knowledge of the training distribution (for instance Corvi and Haywoodsloan) seem to achieve robustness. I would be glad to know the authors thoughts on this regard.

I really like the core idea behind this work personally, and I think adding some predictive value behind these observations would make me view the work in a much more positive light.

References

  1. Rajan, A. S., & Lee, Y. J. (2025). Stay-positive: A case for ignoring real image features in fake image detection. arXiv preprint arXiv:2502.07778.
3 comments

Comment 1 of 4 — Overview and presentation

Thank you for a thoughtful review and for liking the core idea. You raise two things: the writing, and predictive power. We have acted on both; the second is partly achievable, and we say where it is not.

Writing (all four points fixed in the revised PDF). (a) The abstract is rewritten: no control-gating details and far fewer numbers; it now says that most detectors flag many more real photographs as AI-generated than ordinary photographs (up to 80% for one detector), and gives the broader trend. (b) "Elevation" is replaced throughout by plain wording ("higher false-positive rate", "increase"). (c) You are right about the role of the control. It is there to show that a detector is reliable on ordinary photographs, so that failing on FLORIDA is informative. The Introduction and Section 3 now say "reliable" and no longer use "interpretable". (d) Section 4.4 now opens with why the experiment matters (if every detector failed on the same images, one shared cue would explain it and ensembling could not help; otherwise each detector's own training matters) and why we use two metrics (pairwise overlap is easy to read but weak for joint dependence among seven detectors; the vote-count distribution uses all seven).

On Section 4.4's conclusion itself: following another reviewer's request we compared the full vote-count distribution with an independent-error baseline. It is over-dispersed (variance 1.87 vs 1.30 expected, p < 10⁻⁴), so detectors fail on largely different images but not independently; we withdrew the claim of independence.


Comment 2 of 4 — Predictive power: Haywoodsloan, CLIPDet vs UnivFD, and families

We agree the paper did not use what is known about training or architecture to explain failures. We ran two controlled analyses. They give partial answers, and we report where they do not.

CLIPDet vs UnivFD (same recipe, different behavior). The two heads differ in backbone weights (CommonPool vs OpenAI CLIP ViT-L/14), preprocessing (CLIPDet resizes the whole image to 224 px; UnivFD takes a 224 px center crop of the native-resolution image), and training data. We trained probes with identical data, split and calibration on each backbone × preprocessing combination, without hard negatives, and measured the share of held-out FLORIDA images flagged:

Probe featuresFLORIDA-test FPR
CommonPool, resize+crop (CLIPDet's)55.9%
CommonPool, crop only59.4%
OpenAI, resize+crop63.2%
OpenAI, crop only (UnivFD's)68.2%
DINOv2 (non-CLIP)31.8%

For reference the released heads flag 81.2% and 42.4% of the same images. So backbone and preprocessing do not explain the roughly 2× gap between the released heads; in our probes this points to training data or objective (to our understanding, UnivFD's head is trained on GAN fakes and CLIPDet's on latent-diffusion data). We could not vary training data here, so that remains a hypothesis.

Which images fail. We predicted each released detector's per-image error from processing covariates (area, JPEG quality, WebP flag), atypicality (distance to ordinary photographs in a neutral DINOv2 embedding) and taxonomy category (5-fold CV AUROC; 0.5 = chance):

Detectorprocessingatypicalitycategoryall
CLIPDet0.700.650.600.73
Corvi-20230.760.470.560.76
Ateeqq0.660.550.560.64
UnivFD0.610.540.450.59
Organika0.570.550.490.57
AEROBLADE0.520.470.510.52

Predictability is partial and detector-specific: Corvi's failures follow processing (below), CLIPDet's depend on both processing and atypicality, and AEROBLADE's are not predicted by any of these. We do not claim a general predictive theory.

Haywoodsloan. We cannot say what makes it robust. Its model card documents no training data (an AutoTrain fine-tune of its own earlier version), so we cannot exclude exposure to images like ours; it is immune to our processing manipulations; and its errors (4.9%) are too few to model. We state this limitation in the paper rather than offer an explanation we cannot test.

Comment 3 of 4 — Section 4.9: is it a true test of generalization?

You are right that training and test images from one collection can overlap, and the paper did not address it enough. We now report the following checks, and keep one caveat.

  1. Similarity audit. For the CLIP probe we computed each test image's maximum cosine similarity to the 396 training images and evaluated only the least-similar half and quartile. FPR falls from 46% / 44% (no hard negatives) to 1.5% / 3.0% (w = 4).
  2. Source-batch hold-out. FLORIDA ships as four collection parts (PART1–4). We held one out entirely, trained with 396 hard negatives from the other three, and tested only on the held-out part. At w = 4, CLIP falls from 44–69% to 1.7–3.0% FPR on each held-out part; Corvi from 10–42% to 0.4–13.4%.
  3. Category hold-out (already in the paper): leaving a whole taxonomy category out of training still collapses its FPR.
  4. Control for generic real data. Adding the same number of provenance-matched or unmodified ordinary COCO images leaves CLIP's FPR at 41–59%, so the effect is specific to FLORIDA's images, not simply more real supervision.

Caveat. The parts are collection batches, not guaranteed to be source-disjoint, and all of these tests stay within one collection. An independently sourced test set is the clean answer, and we say so in Limitations.


Comment 4 of 4 — Questions 1–3

Q1: Stay-Positive and Corvi⁺. We see a plausible connection. Stay-Positive argues that detectors learn spurious real-image features, so the absence of such features can push a prediction toward "fake". Our spectral analysis found FLORIDA images carry less high-frequency power than ordinary photographs and even than generated images, and AEROBLADE's evidence concentrates in smooth, low-salience regions, which fits that picture. But we cannot test it directly: we could not locate released Corvi⁺ weights or code. As a weak substitute we tried the idea on our own probe (non-negative last layer on Corvi's frozen features, our training data). It did not reduce FLORIDA false positives (40.0% vs 25.6%) and 99.5% of its weights went to zero, so we regard it as inconclusive and not as evidence about their model.

Q2: Corvi and post-processing. We agree, and have supporting evidence. Corvi flags 26.0% of JPEG FLORIDA images but only 1.1% of WebP ones; processing covariates alone predict its failures (AUROC 0.76) while atypicality does not (0.47); and a processing-stress envelope reproduces and exceeds its FLORIDA false-positive rate on ordinary photographs (38% vs 26%). This is consistent with a detector keyed to local compression traces more than to scene content. We have not independently verified the training-crop size you mention, so we do not rely on it: our argument rests only on the measurements above. We run Corvi at full resolution, as in the authors' released configuration. Corvi is not robust in practice, since its FPR is 10× its control rate; the its FPR is 10× its control rate, and our evidence points to processing history rather than scene content, though we cannot rule out a small residual

Q3: Generalizable detectors without knowing the test distribution. We agree that a data fix presupposes awareness of the tail. Two observations bear on how much awareness is needed: hard negatives from other taxonomy categories and from other collection batches still transfer to held-out ones, so the specific kind of fake-looking image need not be known in advance, but some sample of the tail is needed. Corvi is not a counterexample to this (above), and Haywoodsloan is the only detector that is robust, for reasons we cannot establish. Constructing detectors that need no such knowledge is, we agree, the more important goal; our contribution is a benchmark and protocol that can tell whether a candidate achieves it.

Reviewer_H7c2OpenReview ↗

Reviewer Response

I thank the authors for taking the time to provide a rebuttal and fix the writing issues.

Predictability: The recent experiments establish the influence of some factors (post-processing, atypicality etc), however I still feel the paper lacks a general theme behind this phenomenon (when does a detector fail on unnatural real images vs when does it not). The Haywoodsloan not providing documentation is a valid concern, however it does not offer the reader a lot of understanding about the topic as a result. I also believe achieving a more predictive understanding of this would result in a solution which is more generalizable than "training on parts of the FLORIDA dataset".

True Test of Generalization: I feel more confident regarding the generalization, but the doubt still remains as to why any exposure to the FLORIDA dataset is needed for generalization.

As a result, I will be keeping my score.

Reviewer_cBQbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FLORIDA, a dataset of 795 genuine photographs selected for appearing artificial, and evaluates ten synthetic-image detectors. It reports elevated false-positive rates across several detector families and substantial improvements through hard-negative training. The evaluation direction is practically relevant, but stronger evidence is needed to isolate the contribution of atypical content and establish the scope and cost of the proposed remedy.

Critical strengths
  • The paper examines a meaningful failure mode that conventional evaluations on ordinary real photographs may overlook.
  • The evaluation includes useful controls: detector sanity checks, provenance matching, native-threshold results, and an additional-generator check.
  • Hard-negative training produces substantial improvements across two backbone families. Leave-one-category-out evaluation provides additional evidence that the improvement extends beyond categories represented during training.
Critical weaknesses
  1. The matched control does not fully isolate a content-specific effect. Matching JPEG quality and resolution is useful, but leaves differences in source, processing history, and semantic composition uncontrolled. The residual gap between FLORIDA and matched COCO therefore cannot be attributed specifically to “fake-looking” content with sufficient confidence. The interpretation should acknowledge these remaining confounders.

  2. The contribution and cost of hard-negative training remain insufficiently established. Equal-count, equal-weight additions of provenance-matched ordinary real images are missing, leaving additional real-image supervision and source adaptation as alternative explanations for the improvement. Furthermore, maintaining performance on the existing control set does not establish that the intervention preserves detection of realistic generated images. Evaluation before and after augmentation on a challenging synthetic-image source such as Chameleon would directly strengthen the negligible-cost claim.

  3. Coverage of recent detectors is limited. The academic detector suite is concentrated in 2023–2024 and omits relevant published methods, including AIDE, which the manuscript already cites, and approaches explicitly addressing content bias such as SFLD. The observed failures remain valid for the evaluated models, but the broader characterization of the current detector ecosystem requires qualification.

  4. Generator coverage limits the scope of the control-based evaluation. The main synthetic control uses SD1.x and DALL·E 3, while the useful SDXL extension still covers a 2023-era model. These sources determine both threshold calibration and detector eligibility through the AUROC sanity check. This does not invalidate the observed false positives at fixed thresholds, but limits how broadly the selected operating points and characterization of detector competence can be generalized.

Questions for authors
  1. Could the authors clarify whether differences in image source and processing history were controlled for beyond JPEG quality and resolution? Any remaining confounders should be acknowledged when interpreting the residual gap as a content-specific effect.

  2. For the CLIP-based probe, could the authors provide one equal-count, equal-weight provenance-matched ordinary-real augmentation baseline? An evaluation of the existing probes on Chameleon before and after FLORIDA augmentation, under the same calibration protocol, would also help assess whether reducing false positives increases missed detections of realistic generated images.

  3. What criteria guided the authors’ selection of detectors, and why were recent methods such as AIDE and SFLD omitted, given their relevance to challenging synthetic images and content bias?

  4. Could the authors indicate whether they plan to expand the control set to include more recent image generators beyond SD1.x, DALL·E 3, and SDXL?

5 comments

Comment 1 of 4 — Overview

We thank the reviewer for a precise and constructive review; all four weaknesses are fair and we have acted on each.

  • Confounders (Q1): we now acknowledge them explicitly, and we withdrew the "content" interpretation of the residual (details in Comment 2).
  • Equal-count baseline (Q2): new result. Matching the count and weight of the added reals with ordinary COCO images does not reproduce the gain (Comment 3).
  • Chameleon (Q2): not yet run — the data is distributed on request by its authors. The evaluation script is written and tested on SDXL.
  • Detectors and generators (Q3, Q4): we qualify the scope of our claims in the revised PDF and give the position on AIDE and SFLD (Comment 4).

A revised PDF is uploaded (abstract, §4.3, §4.4, hard-negative paragraph, Limitations, new appendix subsections).


Comment 2 of 4 — Q1: Confounders behind the residual gap

You are right that quality and resolution were the only things matched, and that source, processing history and semantic composition were not. We have done three things.

1. Tightened the matching. 189 of 795 FLORIDA files are WebP despite .jpg/.jpeg names (a bug in our matching, which could only read JPEG quality), so we now compare the 606 true JPEGs against the matched control. Adjusting for log-area and quality in a logistic regression leaves the odds ratios unchanged (e.g. CLIPDet 21.9 [16.4, 30.1], Ateeqq 4.8 [3.4, 7.0]).

2. Stress-tested processing history. A single re-encode is a weak control, so we built three harsher histories of the same COCO images (double JPEG; down-then-up resize chain; sharpen + lower quality) and took the per-image maximum as a deliberately generous "processing explains it" envelope. A resize chain alone lifts CLIPDet on ordinary photos from 14.6% to 49.8%.

DetectormatchedenvelopeFLORIDA-JPEGresidual vs envelope
CLIPDet14.6%54.6%75.7%1.4× (p < 10⁻¹²)
Ateeqq10.2%17.8%32.8%1.8× (p < 10⁻⁸)
Corvi-202319.8%38.3%26.0%0.7× (n.s.)
UnivFD29.2%54.0%40.6%0.8× (n.s.)
Organika31.4%42.6%39.1%0.9× (n.s.)
AEROBLADE32.8%50.8%36.8%0.7× (n.s.)
Haywoodsloan4.6%4.8%4.8%1.0× (n.s.)

For four detectors, plausible processing alone reproduces or exceeds FLORIDA's FPR; only CLIPDet and Ateeqq keep a significant residual, and it is much smaller than a single re-encode suggested (5.2× and 3.2×).

3. Changed the interpretation. We no longer call the residual "content-specific". It is a non-compression/resolution residual that can include photographic style, post-processing, acquisition pipelines and semantic composition — FLORIDA and COCO differ in all of these by construction. A source we did not model (e.g. social-platform processing) could still contribute. The decisive control, ordinary photographs from FLORIDA's own venues, remains future work and is now stated in §4.3 and Limitations.

Comment 3 of 4 — Q2: Equal-count baseline, preservation of detection, Chameleon

Equal-count, equal-weight baseline. We repeated the hard-negative experiment with the same split, probes, calibration and weight, replacing the 396 FLORIDA-train images by 396 fresh COCO images (disjoint from the control and fine-tuning pools): either provenance-matched (re-encoded to the area and JPEG quality of FLORIDA-train JPEGs) or unmodified. FLORIDA-test FPR (%), no added negatives = 55.9 (CLIP) / 25.6 (Corvi):

Added real negativesCLIP w=1CLIP w=4Corvi w=1Corvi w=4
FLORIDA-train (n=396/394)4.81.05.03.3
provenance-matched COCO (396/392)52.641.115.816.1
unmodified COCO (396/396)63.959.127.628.6

Extra real supervision alone does nothing; provenance adaptation helps only partly (more for Corvi, whose failures are largely explained by processing history in our analysis, little for CLIP); neither approaches the FLORIDA-train result. Caveat: this controls compression and resolution, not source or photographic style, so it cannot say whether the gain comes from atypical content or from FLORIDA's source — we state this in the appendix. An independently sourced evaluation set would separate them.

Detection of realistic generated images. The control fakes are the same generators as the probes' training fakes, so you are right that unchanged control AUROC did not establish preservation. On 250 SDXL images (absent from all training), TPR at the control-calibrated threshold is unchanged: CLIP probe 97.6% at w=0 and w=4 (98.8% at w=1); Corvi probe 99.6% throughout; SDXL-vs-COCO AUROC is also unchanged. SDXL is not a hard case, though.

Chameleon. We have written and tested the evaluation: thresholds are calibrated on our control set only, never on Chameleon; it reports AUROC, TPR on fakes and FPR on reals at w = 0, 1, 4 before/after FLORIDA augmentation. We will report the result in the paper only if the data is obtained. Until then, the paper's "negligible cost" is qualified to "on the control set and on SDXL".


Comment 4 of 4 — Q3 and Q4: Detector and generator coverage

Detector selection (Q3). The criterion stated in §3.2 was publicly released weights with the authors' own test-time pipeline, covering six families (CNN-artifact, upsampling, frequency, CLIP-feature, diffusion-reconstruction, deployed classifiers).
AIDE and SFLD were not included in the original suite. We have since tried to add them: AIDE's code is public but its checkpoints are on Google Drive, which we could not access during this work; for SFLD we found no public code or weights. A reimplementation would not be the published detector, so we would not report one as such.

We agree that the suite stops at 2024 and the characterization needs qualifying. The revised PDF now says "ten public detectors (2020–2024)" in the contributions and conclusion instead of "the current detector ecosystem", and Limitations names the omission of AIDE and SFLD. The observed failures are, as you note, valid for the evaluated models.

Generator coverage (Q4). The reviewer is right that SD 1.x, DALL·E 3 and SDXL all date from 2022–2023 and set both thresholds and detector eligibility. Two points limit the impact. First, the paper's native-threshold results (see the native-threshold appendix) do not depend on our control generators; the elevated FPR is also visible there. Second, adding SDXL and recalibrating changed neither the sanity-check outcome nor the main result (Appendix on SDXL). We have added to Limitations that operating points and competence judgments may not transfer to newer generators. We do not plan to expand the control set within this discussion period; extending threshold calibration to a newer generator (e.g. a DiT or flow-matching model) is the first item in our future work.

Reviewer_cBQbOpenReview ↗

Request to Complete the Author Responses

Dear Authors,

Thank you for your responses. Several sections still contain unresolved “[AUTHOR: …]” placeholders. Please replace these with the relevant factual information, explicitly noting any information or results that are not yet available, so that I can fully assess how my concerns have been addressed.

Reviewer_mVp8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper build a dataset named FLORIDA which including 795 curated photographs. This dataset intended to be genuine but visually fake-looking, and evaluates ten synthetic-image detectors. Seven detectors pass a control-benchmark performance gate, and six exhibit significantly higher false-positive rates on FLORIDA than on ordinary photographs.

The paper examines compression and resolution effects, cross-detector error overlap, image categories, frequency statistics, and spatial attribution. It also reports that adding several hundred FLORIDA images as real hard negatives substantially reduces false positives on held-out images, with additional experiments on a second backbone and unseen taxonomy categories. It also provides comparison of human judgments and a vision–language model baseline.

Critical strengths
  1. The paper studies a consequential failure mode: synthetic-image detectors falsely flagging genuine but unusual photographs.

  2. The evaluation covers diverse detectors and includes useful controls and training experiments, providing evidence of both the problem and a potential remedy.

  3. This paper provides comparison of human judgments and a vision–language model baseline.

Critical weaknesses
  1. The dataset contains only 795 images, with some categories represented by very few examples. This limits the strength of category-level conclusions and evidence for generalization beyond this collection.

  2. The contribution is primarily an empirical benchmark and a set of experimental observations, with limited methodological novelty. Several broader explanations remain hypotheses rather than firmly established findings.

  3. Authenticity verification is insufficiently documented, despite being essential to interpreting detector outputs as false positives.

  4. Compression/resolution matching does not isolate a causal content effect, and weak pairwise error correlations do not establish independent failure mechanisms.

Questions for authors
  1. How was each photograph’s authenticity verified, and can source-level provenance records be provided?

  2. How can the residual difference after compression/resolution matching be distinguished from other source or processing effects?

  3. Do the independence conclusions hold when comparing the full detector vote-count distribution against an independent-error baseline?

  4. Does hard-negative training preserve detection performance on unseen generators and reduce false positives on independently sourced photographs?

5 comments

Comment 1 of 5 — Overview

We thank the reviewer. Your questions led to four new analyses, and two of them changed claims in our paper; we report that openly.

  1. Independence (Q3): you were right. Vote counts are over-dispersed relative to independent errors (variance ratio 1.45, p < 10⁻⁴), so we withdraw "no shared cue drives the failures".
  2. Matching (Q2): 189 of 795 files are WebP, not JPEG, so our matching covered only the 606 JPEGs. Redone on JPEGs, with covariate adjustment and a processing-stress envelope, a residual survives only for CLIPDet (≈1.4×) and Ateeqq (≈1.8×), down from 5.5× and 3.1×.
  3. Hard negatives (Q4): detection of an unseen generator (SDXL) is preserved and the gain does not rest on near-duplicates. An independently sourced set is untested.
  4. Authenticity (Q1): all 795 images were checked by reverse-image search. A per-image provenance spreadsheet (source link or origin, where found) will accompany the dataset release.

Weakness 1 (size). 795 images limits generalization, and the paper says so. Category-level tests use only the seven categories with n ≥ 20 (779 of 793 images); the headline claim (six of seven detectors have significantly higher FPR, p < 10⁻⁴) is a whole-dataset comparison.

Weakness 2 (novelty). This is an empirical benchmark paper. Explanations that were hypotheses are now labeled as such. The contribution is the calibrated protocol, the decomposition of how much failure is processing history, and a hard-negative fix on two unrelated backbones.

A revised PDF is uploaded.


Comment 2 of 5 — Q1: Authenticity verification and provenance records

Verification. Two independent annotators checked every one of the 795 images by Google reverse-image search, and agreed on 767 (96.5%); disagreements were resolved by discussion. An image was accepted if the search found an original source (photographer, agency or publication) or a posting that predates widely available text-to-image generators. 740 images (tier A) have a traceable original source; 55 (tier B) have only an early-dated posting; none were left unresolved. Reverse search shows where and when an image appeared, not that it is unedited, so tier B is weaker evidence and we report it separately. A per-image spreadsheet (source link and tier) will accompany the dataset release and the supplementary material.

Robustness to residual error. Reverse search can miss a fake, so we also bound how much hidden synthetic contamination would overturn each result, assuming the worst case (every contaminating image is flagged). The number of FLORIDA images that would have to be synthetic is: CLIPDet 627 of 793 (79%), Ateeqq 188 (24%), UnivFD 185 (23%), Organika 253 (32%), AEROBLADE 124 (16%), Corvi-2023 134 (17%). Discarding the 75 images flagged by at least five of seven detectors leaves all six detectors significant (ratios 1.5–32.7×). These are sensitivity bounds, not proof of authenticity. A contaminated subset would also produce part of the over-dispersion in Comment 4, which the paper now says.

Metadata. 0 of 795 files carry a generator marker (weak evidence, since web re-encoding strips metadata). Dropping the 64 files that name an image editor changes no conclusion (CLIPDet 80.0% → 80.7%).

Comment 3 of 5 — Q2: Residual after compression/resolution matching

We cannot claim to isolate "semantic content" — that was over-reach in our wording, and we will rename the residual "non-compression/resolution residual" (it can include photographic style, post-processing, and acquisition pipelines, which are part of what makes these photos look fake). What we can do is bound how much of it processing history could explain. New analyses:

(a) A correction. 189/795 files are WebP with .jpg/.jpeg names; our matching read JPEG quantization tables and silently covered only true JPEGs. We now split by true codec. On the valid comparison (606 JPEG FLORIDA vs JPEG-matched COCO), the residual FPR ratios are: CLIPDet 5.2×, Ateeqq 3.2×, UnivFD 1.4×, Corvi 1.3× (p = 0.01), Organika 1.2× (p = 0.005), AEROBLADE 1.1× (n.s.), Haywoodsloan 1.0× (n.s.). Our earlier statement that provenance explains all of Corvi's elevation was an artifact of pooling (Corvi fires on 26.0% of JPEG FLORIDA but 1.1% of WebP FLORIDA).

(b) Covariate adjustment. Matching was marginal, not joint. Pooled logistic regression (FLORIDA-JPEG ∪ matched COCO) with log-area and JPEG quality as covariates gives adjusted odds ratios essentially equal to the raw ones — CLIPDet 21.9 [16.4, 30.1], Ateeqq 4.8 [3.4, 7.0], UnivFD 1.7, Organika 1.4, Corvi 1.4 [1.07, 1.92], AEROBLADE 1.2 [0.96, 1.55], Haywoodsloan 1.05 [0.61, 1.87]. Within FLORIDA, resolution and quality alone predict detector errors weakly (cross-validated AUROC 0.46–0.66).

(c) A processing-stress envelope. A single re-encode is a weak control, so we built three harsher histories of the same 500 COCO images — double JPEG, down-then-up resize chain, and unsharp-mask + lower quality — and took the per-image maximum over {matched, three stress variants} as an adversarial bound on "processing explains it". The simple resize chain alone raises CLIPDet's FPR on ordinary photos from 14.6% to 49.8%, so processing matters far more than our single-pass control showed.

Detectormatchedworst single variantenvelope (max)FLORIDA-JPEG
CLIPDet14.6%49.8%54.6%75.7%
Ateeqq10.2%15.6%17.8%32.8%
Corvi-202319.8%30.9%38.3%26.0%
UnivFD29.2%51.6%54.0%40.6%
Organika31.4%35.8%42.6%39.1%
AEROBLADE32.8%48.0%50.8%36.8%
Haywoodsloan4.6%4.2%4.8%4.8%

Reading. For Corvi, UnivFD, Organika and AEROBLADE, plausible processing reproduces or exceeds FLORIDA's FPR — for these the elevation is consistent with a provenance shortcut. For CLIPDet and Ateeqq a residual survives even the adversarial envelope (1.4×, p = 10⁻¹³; 1.8×, p = 10⁻⁸). The envelope is deliberately generous to the processing explanation, so these are conservative lower bounds. What we cannot exclude is an unmeasured processing history (e.g. social-platform pipelines); an independent sample of ordinary photos from the same venues would be the decisive control, and we list it as the key open limitation. We will revise §4.3, the abstract and Limitations accordingly.

Comment 4 of 5 — Q3: Vote-count distribution vs independent-error baseline

Thank you — this was the right test, and it contradicts our wording. We compared the observed distribution of the number of detectors (of 7) that flag each image (n = 793) to the Poisson-binomial distribution implied by each detector's marginal error rate under independence, and by 20,000 permutations (each detector's error column permuted independently; marginals preserved exactly).

detectors fooled k01234567
observed3615122618911660150
expected if independent18.0122.3252.6243.0121.831.53.70.1

Variance of the count: 1.87 observed vs 1.30 expected (ratio 1.45; permutation p < 10⁻⁴; χ² goodness of fit 84.3, 5 d.o.f.). The excess is at both tails — twice as many images fool no detector, and ~4× as many fool six. Over-dispersion persists when any single detector is dropped (ratio 1.27–1.45; AEROBLADE is the most influential). Consistently, 18/21 pairwise φ are positive (mean 0.074, median 0.037; the paper reported only the median).

Conclusion. Pairwise correlations are individually weak but jointly reflect a shared per-image factor — a common difficulty or, in part, a few hidden synthetic images (see our answer on authenticity, which bounds this). We withdraw "no shared cue drives the failures" and replace it with: detectors fail on largely different images but not independently — a common difficulty factor exists, and 'which cue' remains open. The practical point survives (any-fires ensembles flag 95.5% of the tail; unanimity flags almost none), but note "0 images fool all 7" is uninformative since independence also predicts ≈0.1.


Comment 5 of 5 — Q4: Hard-negative training: unseen generators and independent photographs

Unseen generators. You are right that our control fakes came from the same generators as the probe's training fakes (SD 1.x, DALL·E 3), so "control AUROC unchanged" did not test this. We evaluated both probes (CLIP ViT-L/14 and Corvi ResNet-50) on 250 SDXL-base-1.0 images never used in any training, at the control-calibrated threshold:

w=0 (no hard neg.)w=1w=4
CLIP probe — SDXL TPR97.6%98.8%97.6%
CLIP probe — FLORIDA-test FPR55.9%4.8%1.0%
Corvi probe — SDXL TPR99.6%99.6%99.6%
Corvi probe — FLORIDA-test FPR25.6%5.0%3.3%

SDXL-vs-COCO AUROC is unchanged (CLIP 0.9995 → 0.9993; Corvi 0.9991 → 0.9998). Caveat: SDXL is a latent-diffusion model like SD 1.x; GAN, autoregressive and flow/DiT generators remain untested, so this shows "no degradation on a newer diffusion generator", not generator-agnostic preservation.

Leakage within FLORIDA. Train/test halves come from the same sources, so near-duplicates could inflate transfer. For the CLIP probe we measured each test image's maximum cosine similarity to FLORIDA-train and evaluated the least-similar half/quartile: FPR falls from 46%/44% (w=0) to 1.5%/3.0% (w=4). (Corvi's pooled post-ReLU features give no usable similarity measure, so we report this audit for CLIP only.) The leave-one-category-out result (Appendix) points the same way.

Independently sourced photographs. Not yet tested; we state this as a limitation and temper "the fix is data" to "hard negatives from the same collection transfer to held-out and category-held-out images".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.