acceptodds
Under review as a conference paper at ICLR 2027

How Do MLLMs Detect AI-Generated Facial Images? A Study of Cross-Source Detection, Provenance, and Explanations

Abstract

Multimodal large language models (MLLMs) are increasingly used to detect AI-generated facial images and provide natural-language explanations. However, it remains unclear which signals influence these judgments and whether the visual evidence reported is reliable across repeated judgments. In this paper, we construct a 4,000-image benchmark covering AI-generated facial images from seven generation sources and real photographs, then evaluate five leading MLLMs in a cross-source setting. We first use full cross-source evaluation and fixed-effects analysis to examine whether detectors more readily identify images produced by models from the same developer. Provenance effects are further investigated by removing C2PA records and applying MarkNull for SynthID removal, while measuring the pixel-level changes. Finally, we assess judgment repeatability and the image support of model explanations. Across the evaluated systems, we find no general detection advantage for same-developer pairs, although effects vary across model combinations. Our provenance interventions further show that C2PA removal does not significantly change overall judgments, whereas SynthID removal produces detector- and source-dependent changes in detection behaviors. Notably, Gemini's detection rate decreases for Gemini Image outputs but increases for GPT Image 2 outputs. For explanations from correct detections, the corresponding images support most reviewed observations, although classification outcomes vary across repeated judgments. Our findings provide a clearer basis for evaluating cross-source MLLM detection and interpreting the visual evidence reported in model explanations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, AI-Generated Image Detection, Cross-Source Detection, Image Provenance, Explanation Reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LyTJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors evaluate five MLLMs on 3,500 generated face images from seven sources and 500 real photographs. They compare detection across developers, measure the effects of provenance processing, and check whether judgments repeat and explanations match the images. They find no consistent advantage for images from the same developer. Removing C2PA records has no significant overall effect after correcting for multiple comparisons. MarkNull removes SynthID and changes detection rates, although it also changes the images. Repeated calls on 200 generated images often produce different labels. Human reviewers find that most observations in explanations of correct detections match the images. The study also includes uncertainty estimates and a processing control using real photographs.

Critical strengths
  1. All five detectors judge the same images, the generators receive the same prompts, and each source contributes the same number of images. The shared real photographs also allow a fair comparison of false positives. Section 4.1 reports that detection rates for pairs from the same developer are only 0.36 percentage points lower than for the other pairs. The 95% confidence interval runs from -2.74 to 1.92 percentage points. This supports the conclusion that there is no consistent advantage across the models tested.

  2. The study identifies problems that a single accuracy score would miss. Across the 200 original images from the two sources used for repeated evaluation, 30.3% of detector and image groups change labels across three calls. Table 4 also shows that Seed repeats the wrong label on 99% of those images. The provenance checks and real-photograph control add another useful finding. Processing can change judgments even on images that started as real photographs. These results matter for anyone considering MLLMs as authenticity judges.

Critical weaknesses
  1. The provenance results need to be read as effects of the processing procedure. MarkNull changes the pixels as well as removing the watermark. Table 14 shows that it raises FAKE rates on real photographs by 4.33 to 56.67 percentage points across the five detectors. The experiment therefore does not isolate reliance on SynthID. Likewise, the C2PA result depends on whether the answering model receives the credentials, which remains unknown in Appendix C.2. The authors acknowledge these limits. They restrict what can be concluded about provenance use, but do not invalidate the measured changes in detection.

  2. The human study in Section 4.3 and Appendix F.7 covers only explanations of correct detections. Reviewers find that 259 of 288 observations match the image. This is useful evidence about those explanations, but it should not be generalized to wrong answers or taken to show which cues caused a decision. The paper states this restriction clearly. I see it as a limit on the coverage of the study rather than a flaw in the reported result.

Questions for authors
  1. Is there any evidence that C2PA credentials reach the models through the endpoints used for detection? This would help interpret the lack of an overall effect after removal.
  2. How were the 100 explanations selected within each detector and source? Was selection random among eligible correct detections? Appendix F.7 describes the quotas and says selection preceded quality review, but the choice within each group is less clear.
Reviewer_YhZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work evaluates 4,000 facial images and reviews explanations from correct detections.

Critical strengths
  1. Experiments relevant to the SynthID removal is timely. To the best of the reviewer's knowledge, no other art tests same-developer advantage or provenance interventions on MLLM image judgments.
  2. The research questions are new.
  3. The presentation is clear.
Critical weaknesses

W1. Despite rich experiments, the major concern of the paper is the confounded claims. For instance, it is known that models without relevant domain knowledge can yield more unstable decoding results, which explains the LLMs "often change the visual cues they emphasize across repeated judgments." Meanwhile, MarkNull removes the watermark by projecting the image onto the latent manifold of a proxy T2I model. It is known that passing real photos through a latent-diffusion autoencoder makes them look generated. W2. The same-developer problem can also be a non-problem as more T2I model developers enforce proprietary watermarks. Third-party detection will likely overfit on the watermarked image manifold, reducing to a "detect some specific watermarks" model instead of a general-purpose AIGI detection model. W3. The benchmark's scope and reusability are limited. All generated images come from one portrait template varied over three attributes (B.2). All negatives come from FFHQ, which are widely trained on modern T2I and detection models.

Questions for authors

In addition to addressing the weaknesses:

Q1. How often do explanations mention watermarks, credentials or metadata under the three conditions? Q2. What is each detector's recall at a common low FPR (e.g. 5%) from the existing scores?

Reviewer_pvD1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how MLLMs detect AI generated facial images across seven generators and five detectors. It also looks at same developer effects provenance signals and consistency of model explanations.

Critical strengths

The cross source setup is quite comprehensive and the statistical analysis for same developer effects is careful. I also like that they test repeatability and not only raw detection accuracy.

Critical weaknesses

The SynthID experiment is quite confounded since MarkNull changes the actual image a lot. So it is hard to say whether the detection changes come from watermark removal or just image processing artifacts.

Questions for authors

Can you better separate the effect of SynthID removal from the visual changes introduced by MarkNull? This is important for how I interpret the provenance results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.