- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a training-free Forensic Readout Alignment (FRA) method. Instead of updating the backbone of the multimodal large language model (MLLM), FRA only edits the lexical projection weights of real/fake answer tokens to mitigate the misalignment between the pretrained verbalizer and forensic discrimination directions. The forensic decision boundary is derived via Fisher direction estimation and carrier shift calibration. Experiments demonstrate state-of-the-art overall performance across seven cross-generator detection benchmarks, and the proposed method is compatible with multiple MLLM backbones. Finally, FRA retains the original question-answering capability of the model while achieving strong generalization for detection.
Critical strengths
- This work reveals that frozen MLLM representations already contain transferable forensic information, which cannot be fully exploited by the native lexical readout. It identifies the critical phenomenon of forensic verbalizer mismatch.
- A gradient-free FRA algorithm is proposed. It only modifies the weights of real and fake answer tokens and constructs the forensic decision boundary via closed-form estimation, without fine-tuning the backbone network.
- It achieves superior detection performance on multiple cross-generator benchmarks and across various MLLM backbones, with favorable robustness.
- The detection adaptation is decoupled from the native model capability. The modified MLLM fully preserves its original visual question answering generation ability, avoiding performance degradation induced by fine-tuning.
Critical weaknesses
- Although generalization experiments report promising results on multiple benchmarks, these results are obtained under the condition that the training set (especially fake data) covers a broad distribution. There lacks evaluations under fully non-overlapping data regimes. For instance, if the training set exclusively consists of GAN-generated samples from CNNDetection while the test set comprises diffusion-generated content (e.g., SD1.4, GLIDE, Midjourney, etc.), the projection of carriers may no longer approximate constants due to the substantial distribution shift between training and test domains, which could shift the decision boundary. This scenario has not been examined.
- Overhead: FRA requires pre-extraction of training-set hidden states for closed-form solving during inference. The additional engineering cost incurred during deployment should be discussed.
Overall, I consider this paper to feature a novel idea and sufficient technical contributions. Its merits outweigh its limitations, and I therefore assign a positive score.
Questions for authors
-
The statement "2,500 class-balanced training images from GenImage (Zhu et al., 2023), with Stable Diffusion v1.4 (SD1.4) as the training generator" does not clearly specify the total volume and composition of the training dataset. Moreover, it remains ambiguous which test sets are fully disjoint from the training set, i.e., whether both real and fake test data have no distribution overlap with the training set.
-
Why does GLM-4.6V-Flash in Table 7 yield 0% accuracy on EvalGEN, GPTImage2, MNW, and T2ICoReBench, which indicates complete label inversion? If so, simply mirroring the decision boundary should yield near-100% accuracy. However, the reported FRA only achieves an adjusted accuracy of 71.23% on GPTImage2.
-
The current evaluation is limited to global binary detection of AI-generated images. For composite samples containing both authentic regions and AI-manipulated regions within a single image, how does FRA make predictions? Accordingly, the definition of whether a sample is regarded as real or AI-generated requires supplementary discussion.
3 comments
Response to Reviewer yydw — Part 2/3
Q3. Training data volume and composition, and verification of test-set disjointness.
R3. Thanks for raising this question. Training uses 2,500 GenImage SD1.4 images (1,250 real ImageNet + 1,250 SD1.4 fakes; Section 4.1, Appendix A.3). The readout is solved from this set alone, with no test data in estimation or tuning. We verify independence at two levels in Table d.
- Image level: zero overlap. All real test images come from COCO, Unsplash or other non-ImageNet sources, and every benchmark synthesizes its own fakes from its own prompts — including DRCT2M, whose SD1.4 images come from COCO captions.
- Generator level: at most two residual overlaps. Only DRCT2M reuses SD1.4 among its 16 generators. Chameleon does not disclose the base models of its SD + LoRA images, so an SD1.x base cannot be excluded. The other five benchmarks are fully disjoint in both real and fake data.
Table h: Data sources and overlap analysis. "-": class absent.
| Dataset | #Real | #Fake | Real source | Fake generators | Image overlap | Generator overlap |
|---|---|---|---|---|---|---|
| GenImage SD1.4 (training) | 1,250 | 1,250 | ImageNet | SD1.4 | - | - |
| AIGINow | 18,000 | 18,000 | COCO (test) | Nano Banana, GPT-4o, Jimeng, Kling, MiniMax, FLUX (Pro/Krea/Dev/Kontext) | None | None |
| DRCT2M | 5,000 | 80,000 | COCO 2017 | 16 SD-family models (LDM, SD1.4/1.5/2.1, SDXL, Turbo, LCM, ControlNet, inpainting) | None | SD1.4 |
| EvalGEN | - | 4,425 | - | FLUX, GoT, Infinity, OmniGen, NOVA | None | None |
| GPTImage2 | - | 10,217 | - | GPT-image-2 | None | None |
| MNW | - | 16,250 | - | Midjourney, DALL-E, GPT-image, Imagen, Firefly, Ideogram, Seedream, FLUX, SD3/SDXL | None | None |
| T2ICoReBench | - | 172,800 | - | 40 diffusion, AR, unified and closed-source models (FLUX, SD3.5, Qwen-Image, Infinity, Janus-Pro, BAGEL, GPT-Image, Nano Banana, Seedream) | None | None |
| Chameleon | 14,863 | 11,170 | Unsplash | Midjourney, DALL-E 3, SD + LoRA | None | Unverified |
Q4. Interpreting GLM's near-zero scores and the limits of boundary mirroring.
R4. Thanks for pointing this out. The zero entries reflect a one-sided collapse of the original verbalizer, not an invertible boundary.
- What 0% means. EvalGEN, GPTImage2, MNW and T2ICoReBench contain only fakes, so accuracy equals fake recall (Section 4.1): a near-zero value means the readout calls almost everything real, not that it separates classes with an inverted sign.
- Why mirroring does not help. It merely turns a constant real predictor into a constant fake one, lifting fake-only subsets to 100% without any discriminative signal. On the three mixed-class benchmarks GLM sits at chance (50.03/50.01/50.01) and stays there after mirroring (49.97/49.99/49.99).
- What FRA reports. The 71.23% on GPTImage2 is the fake recall of the source-fitted readout, with no target-specific mirroring or recalibration. The same fixed readout stays well above chance on all mixed-class benchmarks (75.89% Chameleon to 96.00% AIGINow).
Table i: GLM-4.6V-Flash on all seven benchmarks. Mirrored = 100 - Frozen.
| Benchmark | Metric (%) | Frozen | Mirrored | FRA |
|---|---|---|---|---|
| AIGINow | Real/fake ACC | 50.03 | 49.97 | 96.00 |
| DRCT2M | Real/fake ACC | 50.01 | 49.99 | 83.88 |
| Chameleon | Real/fake ACC | 50.01 | 49.99 | 75.89 |
| EvalGEN | Fake recall | 0.00 | 100.00 | 95.45 |
| GPTImage2 | Fake recall | 0.00 | 100.00 | 71.23 |
| MNW | Fake recall | 0.00 | 100.00 | 77.54 |
| T2ICoReBench | Fake recall | 0.01 | 99.99 | 97.73 |
Response to Reviewer yydw — Part 3/3
Q5. Behavior on partially manipulated images and the definition of the real/fake label.
R5. Thanks for this insightful comment. We clarify the label definition and FRA's prediction mode, and add an evaluation on locally edited images.
- Label definition. Benchmarks label an image fake if any part of it was produced or altered by a generative model, regardless of the edited area. This underlies DRCT2M's inpainting variants and OpenSDID's local-editing splits, and we follow it.
- How FRA predicts. The readout acts on the single final-layer hidden state of the whole image, so FRA gives an image-level verdict without localization: a partial edit is detected only if it shifts the global representation along the forensic direction.
- Additional evaluation. On two class-balanced OpenSDID subsets of the OpenMMSec OOD split (FLUX.1 and SD family, 250 native real + 250 locally edited each, edits verified against the provided masks, no extra training or target calibration), FRA attains 75.40 mean accuracy, leading the best matched-training method by 16.1 points and the best released checkpoint (B-Free) by 7.9 points.
Table j: Locally edited image detection (Acc %), Matched (our 1,250 ImageNet real + 1,250 SD1.4 fake) / Released (official checkpoints).
| Method | FLUX | SD family | Mean |
|---|---|---|---|
| CNNSpot | 53.80 / 50.00 | 49.80 / 50.00 | 51.80 / 50.00 |
| UnivFD | 62.00 / 53.00 | 56.60 / 52.20 | 59.30 / 52.60 |
| FatFormer | 59.40 / 50.00 | 56.00 / 51.60 | 57.70 / 50.80 |
| AIDE | 52.20 / 50.40 | 51.20 / 49.60 | 51.70 / 50.00 |
| DRCT | 52.60 / 51.20 | 52.00 / 59.00 | 52.30 / 55.10 |
| Effort | 56.40 / 50.20 | 53.40 / 51.60 | 54.90 / 50.90 |
| B-Free | 39.20 / 59.40 | 42.00 / 75.60 | 40.60 / 67.50 |
| DDA | 53.40 / 55.80 | 56.60 / 73.20 | 55.00 / 64.50 |
| Deep-VRM | 51.60 / 51.40 | 50.00 / 50.20 | 50.80 / 50.80 |
| GenShield | 50.00 / - | 50.00 / - | 50.00 / - |
| FRA | 71.20 / - | 79.60 / - | 75.40 / - |
Response to Reviewer yydw — Part 1/3
We sincerely thank Reviewer yydw for the thoughtful questions and for recognizing our identification of the forensic verbalizer mismatch, the gradient-free readout edit, the robust cross-generator and cross-backbone gains, and the preserved original question-answering capability. We address the main concerns below.
R1. Thanks for this question. Our default fake training data come from SD1.4 alone, not a broad mixture (Section 4.1). Table 5 further fits FRA on each of eight GenImage generators separately, including GAN-based BigGAN (88.61% mean). We add a strict GAN-to-diffusion test (Table f):
Table f: ProGAN-fitted FRA on unseen diffusion generators. Carrier = mean ± std of v⊤h^.
R2. Thanks for raising this point. We distinguish one-time adaptation from online inference, as summarized in Table g.
FRA thus offers low-cost adaptation with no added inference overhead.
Table g: Adaptation and inference cost against existing methods.