- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a detector for generated-images. The method is about inserting a frozen pretrained LLM layer between a frozen CLIP image encoder and a lightweight classifier. It achieves performance gains (about +3 to +4 accuracy points) and is supported by ablations and visualizations. The claim is that the improvements come from language-derived relational priors rather than generic pretrained-transformer effects.
Critical strengths
The method is simple and practically attractive. One frozen LLM layer, frozen CLIP, and only 4.3M trainable parameters.
Cross-generator results are good under multiple source-target shifts.
In the ablations, there are the interesting comparisons. E.g. capacity-matched MLP, random transformer, class-token-only, and multiple-LLM comparisons.
The paper is transparent about gains and what is hard. It mentions Midjourney, ADM, and VAR as difficult cases; that is honest.
Critical weaknesses
The core idea was already established by Pang et al. (2024): project visual tokens into a frozen pretrained LLM transformer block, remove causal/rotary text assumptions, and use the block as a visual encoder layer. The current paper applies this recipe to generated-image detection, freezes CLIP, adds attention LoRA, and gives task-specific analysis. I wonder whether this is sufficient to be interesting for an ICLR audience. A comparison with a task-adapted Pang-baseline would make the merit of the paper stronger.
The paper mentions a lot of times the importance of relational modeling. I can see that it's important. However, I wonder if the paper really pins down that this it the main driver for the good results. E.g. one ablation changes two things at the same time, making it impossible to distinguish the causing effect. The other ablation with the random transformer shows mainly that pretraining matters, which is not surprising, and not as an argument for relations in (generated) images.
Several related works are absent from the main comparisons. E.g. Community Forensics, AntiFakePrompt, and others. Even if protocols or code availability prevent direct inclusion or comparison, the paper would be stronger with a discussion on qualitative differences and some reimplementation or comparison by other means.
The discussion of CLIP models in the related work makes them appear incapable of relational modeling. That conclusion is too broad. CLIP-based methods may alter token interactions, incorporate local features, etc.
Questions for authors
Why are Community Forensics, bias-free training, Breaking Semantic Artifacts, C2P-CLIP, and other recent detectors not included in the main tables? Which cannot be evaluated under the exact protocols, and why?
Can you include a task-adapted Pang-baseline to evaluate the additions in this paper?
How does performance change under JPEG compression, resizing, cropping, screenshots, blur, and common social-media processing? Can a threshold selected on the source domain transfer under these perturbations?
Please report low-FPR operating points and calibration metrics. Given near-saturated AP but lower fixed-threshold accuracy, how much of the gain is ranking versus calibration?
4 comments
Response to Reviewer PpTn (Part 2/3: Relational Mechanism & Recent Detector Baselines)
W2: Isolating Relational Token Interaction from Patch Information Access
You rightly noted that moving from Row 3 ( only) to Row 4 (all tokens) in Tab. 6 changes two things simultaneously: gains access to patch features and weights them via pretrained content-based attention. To separate these two factors (App. D.2, Tab. 18), we add two intermediate controls: Uniform attention (same pretrained TinyLlama layer and tokens, but attention weights fixed to ) and MLP + patch mean (feeding the concatenation of and the mean patch token into the capacity-matched MLP):
| Re-encoder (Protocol I, ACC / AP %) | Only | + Patch Tokens, Uniform Weights | + Patch Tokens, Content Attention |
|---|---|---|---|
| Capacity-matched MLP | 85.92 / 96.10 | 87.32 / 97.80 | — |
| Pretrained TinyLlama layer | 88.95 / 98.54 | 89.57 / 98.68 | 91.55 / 99.38 |
Within the ACC / AP step of Tab. 6 (Row 3$\to$4), uniform access to all patches accounts for only ACC / AP, whereas content-based pretrained attention contributes ACC (76% of the step) and AP (83%). Giving patch means to the MLP reaches only 87.32% ACC.
We also fully agree that cross-token interaction is not the sole driver of the total ACC gain over linear probing: token-wise re-encoding of contributes ACC, uniform patch access ACC, and content-based attention ACC. Notably, both components depend on language pretraining: in App. D.1 (Tab. 16), adding patch interaction lowers AP for DINOv2-g () and MetaCLIP-H () but raises AP across all three language blocks ( to ). We have revised Secs. 1 and 3.2 to precisely define relational modeling in our single-layer readout (Eq. 4) as content-based cross-token evidence aggregation and to state both complementary gains transparently.
W3 & Q1: Comparisons with Recent Detectors and Protocol Compatibility
Using official code, released weights, and default thresholds, we evaluate the four available methods on the Protocol I test manifest (13 generators, real + fake per set; App. E.1, Tab. 22):
| Method | Training Data | Seen P1 Test Generators | Diffusion (7) | GAN (6) | Overall (13) |
|---|---|---|---|---|---|
| Ours | GenImage SDv1.4 | SDv1.4 only | 91.17 / 99.46 | 91.99 / 99.28 | 91.55 / 99.38 |
| C2P-CLIP (AAAI'25) | GenImage SDv1.4 (official) | SDv1.4 only | 96.37 / 99.11 | 69.43 / 80.82 | 83.93 / 90.67 |
| B-Free (arXiv'24) | COCO + SD2.1 inp. (361k) | None | 91.74 / 97.88 | 86.99 / 93.60 | 89.55 / 95.90 |
| AntifakePrompt (arXiv'23) | COCO + SD3/SD2-inp./LaMa (~180k) | None | 81.14 / 90.96 | 66.82 / 88.51 | 74.53 / 89.83 |
| Comm. Forensics (224px) | 4,803 generators (5.4M) | 6 of 13 (+2 same arch.) | 94.42 / 99.04 | 96.68 / 99.82 | 95.47 / 99.40 |
| Comm. Forensics (384px) | 4,803 generators (5.4M) | 6 of 13 (+2 same arch.) | 96.91 / 99.89 | 98.09 / 99.93 | 97.45 / 99.91 |
Protocol compatibility:
- Exact Protocol I:
C2P-CLIPprovides official GenImage SDv1.4 weights; it achieves higher ACC on seen-family diffusion sets (96.37% vs. 91.17%, though lower AP: 99.11% vs. 99.46%) but degrades sharply on unseen GANs (69.43% / 80.82%), trailing our method by 7.62 ACC / 8.71 AP overall. - Data-centric / VQA paradigms (grouped separately): For
B-Free(~300k SD2.1 content-aligned inpainted COCO pairs),AntifakePrompt(multi-source VQA prompt tuning of a 7B VLM), andCommunity Forensics(5.4M images across 4,803 generators), dataset construction is core to the method, so retraining on plain SDv1.4 breaks their design. Trained on a single generator, our model still outperformsB-Free(+2.00 ACC / +3.48 AP) andAntifakePrompt(+17.02 ACC / +9.55 AP).Community Forensicsattains higher overall accuracy because its training set directly includes 6 of the 13 test generators (SDv1.4, SDv1.5, ADM, GLIDE, VQDM, BigGAN), 2 same-architecture families (ProGAN, StyleGAN), and ImageNet real images; on the 5 test generators unseen by its training set (Wukong, Midjourney, CycleGAN, StarGAN, GauGAN), our single-source model matches its 224px ranking performance (99.48% vs. 99.42% AP; 384px: 99.90% AP), trailing only in fixed-threshold ACC (93.28% vs. 95.24%). - Breaking Semantic Artifacts (Zheng et al., 2024): Neither checkpoints, custom DiffusionDB/LAION splits, nor a required model definition file in the public repo are available, preventing faithful numerical reproduction; we discuss it qualitatively in Sec. 2 and App. E.1.
Response to Reviewer PpTn (Part 3/3: Post-Processing Robustness, Low-FPR & Calibration)
Q3: Post-Processing Robustness and Source-Domain Threshold Transfer
We evaluate 5 CLIP-based detectors (all trained with standard blur+JPEG augmentation) across 23 perturbations on all 13 Protocol I sets (Sec. 4.5, Tab. 11; App. F, Tabs. 27–28; setup details in our reply to bPUv):
| Perturbation (13-set ACC / AP %) | Ours | Effort | VIB-Net | UnivFD | C2P-CLIP |
|---|---|---|---|---|---|
| Clean (aug-trained, 1k+1k split) | 86.42 / 97.28 | 85.33 / 96.52 | 88.38 / 95.11 | 65.30 / 85.97 | 83.94 / 90.68 |
| JPEG () | 76.13 / 91.28 | 68.87 / 86.98 | 74.69 / 83.78 | 57.23 / 81.14 | 58.74 / 66.94 |
| Resize (25/50/75%) | 76.72 / 87.42 | 74.08 / 81.18 | 71.02 / 76.92 | 66.21 / 72.53 | 52.28 / 69.19 |
| Blur () | 72.68 / 81.82 | 68.55 / 76.06 | 67.06 / 72.49 | 60.16 / 67.10 | 52.74 / 63.95 |
| Crop (3 settings) | 73.88 / 90.25 | 73.92 / 90.05 | 77.98 / 84.08 | 66.06 / 83.94 | 79.92 / 86.26 |
| Screenshot () | 76.22 / 91.66 | 78.38 / 89.88 | 77.76 / 84.79 | 71.26 / 81.98 | 70.64 / 85.81 |
| Social media | 66.61 / 79.00 | 59.91 / 74.59 | 69.80 / 77.59 | 60.25 / 79.90 | 63.02 / 67.12 |
| Mean over all 23 settings | 75.74 / 88.34 | 73.32 / 84.27 | 75.00 / 81.17 | 61.96 / 76.93 | 64.05 / 74.01 |
Across all 23 perturbations, our detector ranks 1st in both mean AP (88.34% vs. Effort's 84.27%) and mean ACC (75.74% vs. VIB-Net's 75.00%), winning both metrics on JPEG, resize, and blur (+4.30 to +6.24 AP over Effort). While our AP also leads on crop and screenshot scaling, fixed-threshold ACC varies across individual perturbations (e.g., C2P-CLIP on crop, VIB-Net on social media), motivating the threshold-transfer test below.
Threshold transfer on perturbed SDv1.4 validation set ( images) vs. fixed (ACC %, App. F, Tab. 28):
| Policy | Clean | JPEG | Resize | Blur | JPEG$\to$Blur | Noise | Jitter | Social | Crop | Screen. |
|---|---|---|---|---|---|---|---|---|---|---|
| Fixed | 86.42 | 76.13 | 76.72 | 72.68 | 71.81 | 74.20 | 86.15 | 66.61 | 73.88 | 76.22 |
| Source-tuned | 79.66 | 80.58 | 75.98 | 72.14 | 73.31 | 80.23 | 83.42 | 67.58 | 73.13 | 75.52 |
Re-selecting on identically perturbed source images helps when a perturbation shifts all logits unidirectionally (Noise , JPEG ACC), but offers limited benefit otherwise because source SDv1.4 classes are nearly perfectly separated (ACC 100%), leaving a wide margin that weakly constrains cross-domain thresholds.
Q4: Low-FPR Operating Points and Ranking vs. Calibration
In App. E.2 (Tabs. 23–24), we separate per-set ranking (Opt. ACC, TPR@1%/5%FPR) and calibration (ECE, Brier) from deployment threshold transfer (Src-TPR, Src-FPR: locking at 5% FPR on SDv1.4 real validation images across all test sets):
| Method (All 13 / 6 GANs) | Opt. ACC | TPR@1% | TPR@5% | ECE | Brier | Src-TPR | Src-FPR |
|---|---|---|---|---|---|---|---|
| Ours | 97.67 (97.19) | 92.46 (91.67) | 98.09 (97.80) | 10.67 (8.82) | 10.02 (8.14) | 98.76 (99.18) | 12.12 (20.27) |
| Effort | 95.76 (94.14) | 82.16 (73.78) | 93.57 (89.68) | 9.64 (8.90) | 7.88 (7.89) | 98.32 (98.75) | 18.98 (33.37) |
| VIB-Net | 92.20 (89.37) | 70.73 (63.88) | 86.60 (81.27) | 10.29 (10.33) | 9.58 (10.72) | 93.89 (95.10) | 19.71 (35.65) |
| UnivFD | 74.28 (65.11) | 18.98 (1.57) | 35.62 (8.12) | 25.15 (27.35) | 25.76 (31.41) | 56.22 (51.35) | 17.85 (32.25) |
| C2P-CLIP | 88.33 (77.68) | 53.32 (16.52) | 71.40 (40.77) | 4.58 (1.57) | 23.57 (24.34) | 97.42 (97.27) | 31.41 (62.55) |
Removing calibration via Opt. ACC, our method still leads (97.67% vs. Effort's 95.76%). While AP is near-saturated, at 1% FPR our recall exceeds Effort by +10.30 points overall and +17.89 points on unseen GANs (91.67% vs. 73.78%), whereas ECE/Brier match Effort and VIB-Net—confirming our gain comes from superior ranking (especially in the low-FPR tail) rather than calibration. Under the source-locked 5%-FPR threshold, our detector simultaneously achieves the highest Src-TPR (98.76%) and lowest Src-FPR (12.12% vs. 17.85–31.41% for baselines). Notably, realized FPR stays near target (5.1%) across the 7 GenImage sets (ImageNet real) but rises to 20.27% on the 6 GAN sets (LSUN/CelebA/COCO real; 32.25–62.55% for all baselines): source thresholds transfer across generators, but drift across real-image sources.
Reply to Rebuttal
It is a good rebuttal, which addresses my earlier concerns and answers questions. I have also looked at the other reviews, they are positive, and I now agree. I have increased my score.
In the rebuttal, I appreciated the added nuance. The authors discuss the changes relative to the similar prior work Peng et al. (2024). They improved one ablation where two things changed at the same time, now they separated them. A comparison with other works was added. The authors added a discussion on related works that I missed before. They also discuss CLIP models with more nuance.
Response to Reviewer PpTn (Part 1/3: Pang et al. Baselines, Contribution & CLIP Discussion)
We thank Reviewer PpTn for the constructive critique. As your questions span seven new tables, we reply in three parts (Part 1: W1/Q2, W4; Part 2: W2, W3/Q1; Part 3: Q3, Q4). All updates appear in Secs. 1–3, 4.5, and Apps. D–F of the revised PDF (in blue).
W1 & Q2: Task-Adapted Pang et al. (2024) Baselines and Core Contribution
Conceptually, Pang et al. use an LLM block as a general-purpose booster for recognition, where the answer depends on what an image contains. In generated image detection, however, as low-level artifacts fade, the evidence lies increasingly in how the high-level semantic contents of an image relate to one another. This is precisely where frozen CLIP features fall short and what a language-pretrained layer can provide. Image forensics is therefore a natural fit for this idea rather than a routine task transfer.
Methodologically, while we adopt Pang et al.'s removal of causal masking and RoPE to feed visual tokens into an LLM block (credited in Sec. 3.2 and App. A), directly applying Pang et al.'s training paradigm fails to generalize in generated-image detection. In all experiments of Pang et al., the visual backbone is trained jointly with a fully frozen LLM block (either from scratch or by end-to-end fine-tuning of a pretrained encoder, without LoRA). Using the exact same wrapper (
LN→Linear→ LLM block →Linear→ tcls head), we reproduce both Pang et al. settings under Protocol I (App. D.2, Tab. 17):Training from scratch reaches only 63.32% ACC (chance-level 53.78% on unseen GANs). Critically, Pang et al.'s fine-tuned CLIP variant trains 72$\times$ more parameters (307.4M vs. 4.25M) yet trails our method by 7.71 ACC points overall (83.84% vs. 91.55%): it fits the seen diffusion family slightly tighter in AP (99.67% vs. 99.46%) but collapses by −11.99 ACC and −4.10 AP on unseen GANs (80.00% vs. 91.99%).
This contrast highlights a fundamental difference between recognition (Pang et al.'s focus) and forensics: under single-generator binary supervision, updating the visual backbone rapidly overfits source-generator artifacts and content/category biases (Guillaro et al., 2024; Zheng et al., 2024). Freezing CLIP and adapting only attention projections via rank-4 LoRA prevents shortcut collapse and rules out backbone co-adaptation—a confounder Pang et al. explicitly left open. Beyond this necessary design departure, we contribute three forensic analyses absent from prior work:
DINOv2-g) and multimodal (EVA-CLIP-8B,MetaCLIP-H) blocks shows language blocks outperform both by +0.8 to +4.0 ACC and +0.9 to +3.0 AP on identical CLIP tokens.W4: Nuanced Discussion of CLIP-Based Detectors
We agree our original wording was overly broad and have revised Secs. 1, 2, and App. A. CLIP's ViT encoder indeed models token interactions via self-attention, and recent detectors (
Effort,C2P-CLIP,CO-SPY) modify weights or fuse local features. Our revised text scopes the claim accurately: CLIP's global image-text contrastive objective does not explicitly supervise inter-region structural/semantic consistency, and we complement frozen CLIP tokens with an external prior learned from autoregressive sequence modeling.