acceptodds
Under review as a conference paper at ICLR 2027

Language Priors without Language: A Frozen LLM Layer for Generated Image Detection

Abstract

Generated image detection increasingly relies on frozen vision-language models such as CLIP, whose features generalize well across generators but are learned through global image-text alignment and remain limited in modeling semantic relationships among image regions and structural consistency. As modern generators suppress low-level artifacts, high-level cues such as semantic and structural anomalies become decisive. We observe that large language models (LLMs) acquire, through autoregressive pretraining, precisely the ability that CLIP lacks: organizing semantic relationships and contextual dependencies among tokens. We therefore propose to treat a single frozen decoder layer of a pretrained LLM as a semantic relation enhancement module for visual tokens. The layer re-encodes the CLIP tokens within the relational structure learned from language, without any language prompts or textual inputs, while both CLIP and the LLM remain frozen and only about 4.3M parameters are trained. Trained on a single generator, our detector reaches 91.55% accuracy and 99.38% average precision across 13 GAN and diffusion generators, surpassing the strongest prior detector by 3.05 accuracy points, and it generalizes to recently released diffusion and autoregressive generators with 98.43% accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generalization, CLIP, Semantic Relation Modeling, AI-Generated Image Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PpTnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a detector for generated-images. The method is about inserting a frozen pretrained LLM layer between a frozen CLIP image encoder and a lightweight classifier. It achieves performance gains (about +3 to +4 accuracy points) and is supported by ablations and visualizations. The claim is that the improvements come from language-derived relational priors rather than generic pretrained-transformer effects.

Critical strengths

The method is simple and practically attractive. One frozen LLM layer, frozen CLIP, and only 4.3M trainable parameters.

Cross-generator results are good under multiple source-target shifts.

In the ablations, there are the interesting comparisons. E.g. capacity-matched MLP, random transformer, class-token-only, and multiple-LLM comparisons.

The paper is transparent about gains and what is hard. It mentions Midjourney, ADM, and VAR as difficult cases; that is honest.

Critical weaknesses

The core idea was already established by Pang et al. (2024): project visual tokens into a frozen pretrained LLM transformer block, remove causal/rotary text assumptions, and use the block as a visual encoder layer. The current paper applies this recipe to generated-image detection, freezes CLIP, adds attention LoRA, and gives task-specific analysis. I wonder whether this is sufficient to be interesting for an ICLR audience. A comparison with a task-adapted Pang-baseline would make the merit of the paper stronger.

The paper mentions a lot of times the importance of relational modeling. I can see that it's important. However, I wonder if the paper really pins down that this it the main driver for the good results. E.g. one ablation changes two things at the same time, making it impossible to distinguish the causing effect. The other ablation with the random transformer shows mainly that pretraining matters, which is not surprising, and not as an argument for relations in (generated) images.

Several related works are absent from the main comparisons. E.g. Community Forensics, AntiFakePrompt, and others. Even if protocols or code availability prevent direct inclusion or comparison, the paper would be stronger with a discussion on qualitative differences and some reimplementation or comparison by other means.

The discussion of CLIP models in the related work makes them appear incapable of relational modeling. That conclusion is too broad. CLIP-based methods may alter token interactions, incorporate local features, etc.

Questions for authors

Why are Community Forensics, bias-free training, Breaking Semantic Artifacts, C2P-CLIP, and other recent detectors not included in the main tables? Which cannot be evaluated under the exact protocols, and why?

Can you include a task-adapted Pang-baseline to evaluate the additions in this paper?

How does performance change under JPEG compression, resizing, cropping, screenshots, blur, and common social-media processing? Can a threshold selected on the source domain transfer under these perturbations?

Please report low-FPR operating points and calibration metrics. Given near-saturated AP but lower fixed-threshold accuracy, how much of the gain is ranking versus calibration?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PpTn (Part 1/3: Pang et al. Baselines, Contribution & CLIP Discussion)

We thank Reviewer PpTn for the constructive critique. As your questions span seven new tables, we reply in three parts (Part 1: W1/Q2, W4; Part 2: W2, W3/Q1; Part 3: Q3, Q4). All updates appear in Secs. 1–3, 4.5, and Apps. D–F of the revised PDF (in blue).

W1 & Q2: Task-Adapted Pang et al. (2024) Baselines and Core Contribution

Conceptually, Pang et al. use an LLM block as a general-purpose booster for recognition, where the answer depends on what an image contains. In generated image detection, however, as low-level artifacts fade, the evidence lies increasingly in how the high-level semantic contents of an image relate to one another. This is precisely where frozen CLIP features fall short and what a language-pretrained layer can provide. Image forensics is therefore a natural fit for this idea rather than a routine task transfer.

Methodologically, while we adopt Pang et al.'s removal of causal masking and RoPE to feed visual tokens into an LLM block (credited in Sec. 3.2 and App. A), directly applying Pang et al.'s training paradigm fails to generalize in generated-image detection. In all experiments of Pang et al., the visual backbone is trained jointly with a fully frozen LLM block (either from scratch or by end-to-end fine-tuning of a pretrained encoder, without LoRA). Using the exact same wrapper (LN →\to Linear →\to LLM block →\to Linear →\to tcls\boldsymbol{t}_{\text{cls}} head), we reproduce both Pang et al. settings under Protocol I (App. D.2, Tab. 17):

VariantVisual BackboneLLM BlockTotal / TrainableAll 13 SetsDiffusion (7, seen)GAN (6, unseen)
Pang et al. (scratch)ViT-S/16, scratchLLaMA-7B F31F_{31}, frozen227.2M / 24.81M63.32 / 71.3271.49 / 81.3853.78 / 59.57
Pang et al. (CLIP FT)CLIP ViT-L/14, FTTinyLlama F21F_{21}, frozen351.4M / 307.38M83.84 / 97.6087.14 / 99.6780.00 / 95.18
OursCLIP ViT-L/14, frozenTinyLlama F21F_{21}, LoRA351.5M / 4.25M91.55 / 99.3891.17 / 99.4691.99 / 99.28

Training from scratch reaches only 63.32% ACC (chance-level 53.78% on unseen GANs). Critically, Pang et al.'s fine-tuned CLIP variant trains 72$\times$ more parameters (307.4M vs. 4.25M) yet trails our method by 7.71 ACC points overall (83.84% vs. 91.55%): it fits the seen diffusion family slightly tighter in AP (99.67% vs. 99.46%) but collapses by −11.99-11.99 ACC and −4.10-4.10 AP on unseen GANs (80.00% vs. 91.99%).

This contrast highlights a fundamental difference between recognition (Pang et al.'s focus) and forensics: under single-generator binary supervision, updating the visual backbone rapidly overfits source-generator artifacts and content/category biases (Guillaro et al., 2024; Zheng et al., 2024). Freezing CLIP and adapting only attention projections via rank-4 LoRA prevents shortcut collapse and rules out backbone co-adaptation—a confounder Pang et al. explicitly left open. Beyond this necessary design departure, we contribute three forensic analyses absent from prior work:

  1. Language vs. visual/multimodal blocks (App. D.1, Tab. 16): Replacing the LLM block with capacity-matched pretrained visual (DINOv2-g) and multimodal (EVA-CLIP-8B, MetaCLIP-H) blocks shows language blocks outperform both by +0.8 to +4.0 ACC and +0.9 to +3.0 AP on identical CLIP tokens.
  2. Disentangling token re-encoding from content-based attention (Part 2 & App. D.2, Tab. 18): Directly testing Pang et al.'s information-filtering hypothesis via tcls\boldsymbol{t}_{\text{cls}}-only and uniform-attention controls.
  3. Separating high-level plausibility from low-level artifacts (App. D.3, Tabs. 19–21): Low-pass filtering, patch shuffling, and noise-matched semantic inpainting show our LLM block detects semantic incoherence (72% pairwise preference vs. 40% for linear probing) under identical generator traces.

W4: Nuanced Discussion of CLIP-Based Detectors

We agree our original wording was overly broad and have revised Secs. 1, 2, and App. A. CLIP's ViT encoder indeed models token interactions via self-attention, and recent detectors (Effort, C2P-CLIP, CO-SPY) modify weights or fuse local features. Our revised text scopes the claim accurately: CLIP's global image-text contrastive objective does not explicitly supervise inter-region structural/semantic consistency, and we complement frozen CLIP tokens with an external prior learned from autoregressive sequence modeling.

AuthorsOpenReview ↗

Response to Reviewer PpTn (Part 2/3: Relational Mechanism & Recent Detector Baselines)

W2: Isolating Relational Token Interaction from Patch Information Access

You rightly noted that moving from Row 3 (tcls\boldsymbol{t}_{\text{cls}} only) to Row 4 (all N+1N{+}1 tokens) in Tab. 6 changes two things simultaneously: tcls\boldsymbol{t}_{\text{cls}} gains access to patch features and weights them via pretrained content-based attention. To separate these two factors (App. D.2, Tab. 18), we add two intermediate controls: Uniform attention (same pretrained TinyLlama layer and N+1N{+}1 tokens, but attention weights fixed to 1/(N+1)1/(N{+}1)) and MLP + patch mean (feeding the concatenation of tcls\boldsymbol{t}_{\text{cls}} and the mean patch token into the capacity-matched MLP):

Re-encoder (Protocol I, ACC / AP %)tcls\boldsymbol{t}_{\text{cls}} Only+ Patch Tokens, Uniform Weights+ Patch Tokens, Content Attention
Capacity-matched MLP85.92 / 96.1087.32 / 97.80—
Pretrained TinyLlama layer88.95 / 98.5489.57 / 98.6891.55 / 99.38

Within the +2.60+2.60 ACC / +0.84+0.84 AP step of Tab. 6 (Row 3$\to$4), uniform access to all patches accounts for only +0.62+0.62 ACC / +0.14+0.14 AP, whereas content-based pretrained attention contributes +1.98+1.98 ACC (76% of the step) and +0.70+0.70 AP (83%). Giving patch means to the MLP reaches only 87.32% ACC.

We also fully agree that cross-token interaction is not the sole driver of the total +6.80+6.80 ACC gain over linear probing: token-wise re-encoding of tcls\boldsymbol{t}_{\text{cls}} contributes +3.03+3.03 ACC, uniform patch access +0.62+0.62 ACC, and content-based attention +1.98+1.98 ACC. Notably, both components depend on language pretraining: in App. D.1 (Tab. 16), adding patch interaction lowers AP for DINOv2-g (−0.40-0.40) and MetaCLIP-H (−0.51-0.51) but raises AP across all three language blocks (+0.84+0.84 to +1.39+1.39). We have revised Secs. 1 and 3.2 to precisely define relational modeling in our single-layer readout (Eq. 4) as content-based cross-token evidence aggregation and to state both complementary gains transparently.

W3 & Q1: Comparisons with Recent Detectors and Protocol Compatibility

Using official code, released weights, and default thresholds, we evaluate the four available methods on the Protocol I test manifest (13 generators, 1k1\text{k} real + 1k1\text{k} fake per set; App. E.1, Tab. 22):

MethodTraining DataSeen P1 Test GeneratorsDiffusion (7)GAN (6)Overall (13)
OursGenImage SDv1.4SDv1.4 only91.17 / 99.4691.99 / 99.2891.55 / 99.38
C2P-CLIP (AAAI'25)GenImage SDv1.4 (official)SDv1.4 only96.37 / 99.1169.43 / 80.8283.93 / 90.67
B-Free (arXiv'24)COCO + SD2.1 inp. (361k)None91.74 / 97.8886.99 / 93.6089.55 / 95.90
AntifakePrompt (arXiv'23)COCO + SD3/SD2-inp./LaMa (~180k)None81.14 / 90.9666.82 / 88.5174.53 / 89.83
Comm. Forensics (224px)4,803 generators (5.4M)6 of 13 (+2 same arch.)94.42 / 99.0496.68 / 99.8295.47 / 99.40
Comm. Forensics (384px)4,803 generators (5.4M)6 of 13 (+2 same arch.)96.91 / 99.8998.09 / 99.9397.45 / 99.91

Protocol compatibility:

  • Exact Protocol I: C2P-CLIP provides official GenImage SDv1.4 weights; it achieves higher ACC on seen-family diffusion sets (96.37% vs. 91.17%, though lower AP: 99.11% vs. 99.46%) but degrades sharply on unseen GANs (69.43% / 80.82%), trailing our method by 7.62 ACC / 8.71 AP overall.
  • Data-centric / VQA paradigms (grouped separately): For B-Free (~300k SD2.1 content-aligned inpainted COCO pairs), AntifakePrompt (multi-source VQA prompt tuning of a 7B VLM), and Community Forensics (5.4M images across 4,803 generators), dataset construction is core to the method, so retraining on plain SDv1.4 breaks their design. Trained on a single generator, our model still outperforms B-Free (+2.00 ACC / +3.48 AP) and AntifakePrompt (+17.02 ACC / +9.55 AP). Community Forensics attains higher overall accuracy because its training set directly includes 6 of the 13 test generators (SDv1.4, SDv1.5, ADM, GLIDE, VQDM, BigGAN), 2 same-architecture families (ProGAN, StyleGAN), and ImageNet real images; on the 5 test generators unseen by its training set (Wukong, Midjourney, CycleGAN, StarGAN, GauGAN), our single-source model matches its 224px ranking performance (99.48% vs. 99.42% AP; 384px: 99.90% AP), trailing only in fixed-threshold ACC (93.28% vs. 95.24%).
  • Breaking Semantic Artifacts (Zheng et al., 2024): Neither checkpoints, custom DiffusionDB/LAION splits, nor a required model definition file in the public repo are available, preventing faithful numerical reproduction; we discuss it qualitatively in Sec. 2 and App. E.1.
AuthorsOpenReview ↗

Response to Reviewer PpTn (Part 3/3: Post-Processing Robustness, Low-FPR & Calibration)

Q3: Post-Processing Robustness and Source-Domain Threshold Transfer

We evaluate 5 CLIP-based detectors (all trained with standard blur+JPEG augmentation) across 23 perturbations on all 13 Protocol I sets (Sec. 4.5, Tab. 11; App. F, Tabs. 27–28; setup details in our reply to bPUv):

Perturbation (13-set ACC / AP %)OursEffortVIB-NetUnivFDC2P-CLIP
Clean (aug-trained, 1k+1k split)86.42 / 97.2885.33 / 96.5288.38 / 95.1165.30 / 85.9783.94 / 90.68
JPEG (q=90/80/70q{=}90/80/70)76.13 / 91.2868.87 / 86.9874.69 / 83.7857.23 / 81.1458.74 / 66.94
Resize (25/50/75%)76.72 / 87.4274.08 / 81.1871.02 / 76.9266.21 / 72.5352.28 / 69.19
Blur (σ=1/2/3\sigma{=}1/2/3)72.68 / 81.8268.55 / 76.0667.06 / 72.4960.16 / 67.1052.74 / 63.95
Crop (3 settings)73.88 / 90.2573.92 / 90.0577.98 / 84.0866.06 / 83.9479.92 / 86.26
Screenshot (×1.25/×0.8\times 1.25/\times 0.8)76.22 / 91.6678.38 / 89.8877.76 / 84.7971.26 / 81.9870.64 / 85.81
Social media66.61 / 79.0059.91 / 74.5969.80 / 77.5960.25 / 79.9063.02 / 67.12
Mean over all 23 settings75.74 / 88.3473.32 / 84.2775.00 / 81.1761.96 / 76.9364.05 / 74.01

Across all 23 perturbations, our detector ranks 1st in both mean AP (88.34% vs. Effort's 84.27%) and mean ACC (75.74% vs. VIB-Net's 75.00%), winning both metrics on JPEG, resize, and blur (+4.30 to +6.24 AP over Effort). While our AP also leads on crop and screenshot scaling, fixed-threshold ACC varies across individual perturbations (e.g., C2P-CLIP on crop, VIB-Net on social media), motivating the threshold-transfer test below.

Threshold transfer on perturbed SDv1.4 validation set (500+500500{+}500 images) vs. fixed τ=0.5\tau{=}0.5 (ACC %, App. F, Tab. 28):

PolicyCleanJPEGResizeBlurJPEG$\to$BlurNoiseJitterSocialCropScreen.
Fixed τ=0.5\tau{=}0.586.4276.1376.7272.6871.8174.2086.1566.6173.8876.22
Source-tuned τ\tau79.6680.5875.9872.1473.3180.2383.4267.5873.1375.52

Re-selecting τ\tau on identically perturbed source images helps when a perturbation shifts all logits unidirectionally (Noise +6.03+6.03, JPEG +4.45+4.45 ACC), but offers limited benefit otherwise because source SDv1.4 classes are nearly perfectly separated (ACC ≈\approx 100%), leaving a wide margin that weakly constrains cross-domain thresholds.

Q4: Low-FPR Operating Points and Ranking vs. Calibration

In App. E.2 (Tabs. 23–24), we separate per-set ranking (Opt. ACC, TPR@1%/5%FPR) and calibration (ECE, Brier×100\times 100) from deployment threshold transfer (Src-TPR, Src-FPR: locking τ\tau at 5% FPR on SDv1.4 real validation images across all test sets):

Method (All 13 / 6 GANs)Opt. ACC ↑\uparrowTPR@1% ↑\uparrowTPR@5% ↑\uparrowECE ↓\downarrowBrier ↓\downarrowSrc-TPR ↑\uparrowSrc-FPR ↓\downarrow
Ours97.67 (97.19)92.46 (91.67)98.09 (97.80)10.67 (8.82)10.02 (8.14)98.76 (99.18)12.12 (20.27)
Effort95.76 (94.14)82.16 (73.78)93.57 (89.68)9.64 (8.90)7.88 (7.89)98.32 (98.75)18.98 (33.37)
VIB-Net92.20 (89.37)70.73 (63.88)86.60 (81.27)10.29 (10.33)9.58 (10.72)93.89 (95.10)19.71 (35.65)
UnivFD74.28 (65.11)18.98 (1.57)35.62 (8.12)25.15 (27.35)25.76 (31.41)56.22 (51.35)17.85 (32.25)
C2P-CLIP88.33 (77.68)53.32 (16.52)71.40 (40.77)4.58 (1.57)23.57 (24.34)97.42 (97.27)31.41 (62.55)

Removing calibration via Opt. ACC, our method still leads (97.67% vs. Effort's 95.76%). While AP is near-saturated, at 1% FPR our recall exceeds Effort by +10.30 points overall and +17.89 points on unseen GANs (91.67% vs. 73.78%), whereas ECE/Brier match Effort and VIB-Net—confirming our gain comes from superior ranking (especially in the low-FPR tail) rather than calibration. Under the source-locked 5%-FPR threshold, our detector simultaneously achieves the highest Src-TPR (98.76%) and lowest Src-FPR (12.12% vs. 17.85–31.41% for baselines). Notably, realized FPR stays near target (5.1%) across the 7 GenImage sets (ImageNet real) but rises to 20.27% on the 6 GAN sets (LSUN/CelebA/COCO real; 32.25–62.55% for all baselines): source thresholds transfer across generators, but drift across real-image sources.

Reviewer_PpTnOpenReview ↗

Reply to Rebuttal

It is a good rebuttal, which addresses my earlier concerns and answers questions. I have also looked at the other reviews, they are positive, and I now agree. I have increased my score.

In the rebuttal, I appreciated the added nuance. The authors discuss the changes relative to the similar prior work Peng et al. (2024). They improved one ablation where two things changed at the same time, now they separated them. A comparison with other works was added. The authors added a discussion on related works that I missed before. They also discuss CLIP models with more nuance.

Reviewer_CAFCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a generated-image detector that enhances frozen CLIP visual representations using a single pretrained LLM decoder layer without textual inputs. CLIP patch tokens are projected into the LLM hidden space, processed through bidirectional attention, and mapped back for binary classification. The CLIP encoder and pretrained LLM weights remain frozen, while lightweight projection layers, LoRA adapters, and a classification head are trained.

The central hypothesis is that relational representations acquired during language pretraining can improve the modeling of semantic and structural inconsistencies in generated images. Experiments across three generalization protocols demonstrate improvements over existing detectors, achieving 91.55% accuracy on 13 GAN and diffusion generators and 98.43% accuracy on emerging generators. Ablation studies examine the contributions of pretrained weights, token interactions, adaptation strategies, and LLM architectures.

Critical strengths
  1. Effective use of pretrained LLM representations for visual forensics. The paper demonstrates that a single pretrained LLM layer can substantially improve generated-image detection without language prompts or text decoding. The approach is computationally attractive in terms of trainable parameters, and the consistent improvements across different pretrained LLMs suggest that the benefits are not specific to a particular language model. This provides an interesting direction for transferring pretrained language representations to visual discriminative tasks.

  2. Strong cross-generator evaluation and informative ablations. The evaluation considers several training and testing protocols, including generalization between GAN and diffusion generators and transfer to autoregressive generators unseen during training. The ablations comparing pretrained layers, randomly initialized layers, capacity-matched MLPs, and class-token-only processing help identify which components contribute to the observed improvements. These experiments substantially strengthen the empirical contribution.

Critical weaknesses
  1. The contribution of language-specific pretraining remains insufficiently isolated. Although the ablations demonstrate that pretrained LLM weights outperform random initialization and capacity-matched MLPs, they do not conclusively establish that the improvement specifically originates from relational knowledge acquired through language pretraining. In particular, a comparison with a pretrained visual Transformer block or another pretrained attention module that processes the same CLIP patch tokens is missing. The current results support the usefulness of pretrained LLM layers, but they provide weaker evidence for the stronger claim that language-derived relational priors are the essential mechanism.

  2. The evaluation does not establish robustness under realistic image transformations. The experiments focus on clean benchmark images, while generated-image detection in practical settings frequently involves JPEG compression, resizing, cropping, and other post-processing operations. The paper explicitly acknowledges that post-processing robustness has not been evaluated. Since the proposed method is intended to identify high-level inconsistencies rather than low-level generation artifacts, demonstrating whether its improvements persist under such transformations would be important for assessing the practical significance of the contribution.

Questions for authors
  1. Could the authors compare the proposed pretrained LLM layer against a pretrained visual Transformer block or another pretrained attention module with comparable computational capacity and access to the same patch tokens? This would help clarify whether the improvements are attributable specifically to language pretraining or to pretrained attention-based feature refinement more generally.

  2. How does the detector perform under common post-processing transformations, particularly JPEG compression and image resizing? Does the advantage over existing CLIP-based detectors remain consistent when these transformations are applied to unseen-generator test images?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CAFC (Part 1/2: Language vs. Visual/Multimodal Pretrained Blocks)

We thank Reviewer CAFC for recognizing our promising cross-modal transfer direction, parameter efficiency, cross-generator generalization, and controlled ablations. We reply in two parts (Part 1: W1/Q1; Part 2: W2/Q2): pretrained visual/multimodal block comparisons are in App. D.1 (Tab. 16), and post-processing robustness in Sec. 4.5 (Tab. 11) and App. F (Tabs. 25–26) of the revised PDF (in blue).

W1 & Q1: Language-Pretrained Layer vs. Pretrained Visual and Multimodal Transformer Blocks

If our gains came merely from inserting a sufficiently large pretrained attention block, then replacing the LLM layer with a comparably sized pretrained visual or vision-language block, fed the exact same CLIP patch tokens, should work equally well. To test this directly (App. D.1, Tab. 16), we replace the TinyLlama block with pretrained Transformer blocks from pure text (autoregressive and masked LM), image-text contrastive learning, and self-supervised visual pretraining, keeping the surrounding projections, LoRA rank, and readout identical. For each block, we also train a class-token-only (tcls\boldsymbol{t}_{\text{cls}} only) variant. Since a single token cannot attend across patches, the gap (Δ\Delta) strictly isolates the gain from cross-token interaction.

Inserted Block (Protocol I, 13-set mean)Pretraining ObjectiveDim ddBlock ParamsAll Tokens (N+1N{+}1)tcls\boldsymbol{t}_{\text{cls}} OnlyToken Interaction (Δ\Delta)
TinyLlama-1.1B (Ours)Text, autoregressive204844.0M91.55 / 99.3888.95 / 98.54+2.60+2.60 / +0.84+0.84
Llama-3.1-8BText, autoregressive4096218.1M91.22 / 99.4789.43 / 98.45+1.79+1.79 / +1.02+1.02
BERT-largeText, masked LM102412.6M89.21 / 98.9788.81 / 97.58+0.40+0.40 / +1.39+1.39
EVA-CLIP-8BImage-text, contrastive4096234.9M88.40 / 98.0787.98 / 97.56+0.42+0.42 / +0.51+0.51
MetaCLIP-HImage-text, contrastive128019.7M87.60 / 97.3287.87 / 97.83−0.27-0.27 / −0.51-0.51
DINOv2-gImage-only, self-sup.153628.3M87.67 / 96.4786.16 / 96.87+1.51+1.51 / −0.40-0.40

The results divide cleanly by pretraining modality rather than scale or generic attention refinement:

  1. Consistent modality gap at matched capacity: Every language-pretrained block (89.21–91.55% ACC, 98.97–99.47% AP) outperforms every visual and multimodal block (87.60–88.40% ACC, 96.47–98.07% AP). Even at matched width and parameter count—Llama-3.1-8B (d=4096d{=}4096, 218M) vs. EVA-CLIP-8B (d=4096d{=}4096, 235M), or BERT-large (d=1024d{=}1024, 12.6M) vs. MetaCLIP-H (d=1280d{=}1280, 19.7M) and DINOv2-g (d=1536d{=}1536, 28.3M)—language blocks lead by +1.5 to +2.8 ACC and +1.4 to +2.5 AP, whereas visual/multimodal blocks barely exceed a randomly initialized layer (86.72% ACC / 96.93% AP in Tab. 6).
  2. Two-level advantage: Language blocks already re-encode tcls\boldsymbol{t}_{\text{cls}} better (88.81–89.43% vs. 86.16–87.98% ACC), and admitting patch-token interaction consistently boosts AP across all three language blocks (+0.84 to +1.39 AP; +1.79 to +2.60 ACC for autoregressive LLMs), whereas AP stagnates or drops for visual/contrastive blocks (−0.51-0.51 to +0.51+0.51 AP). This confirms that the improvement stems from priors learned in language pretraining rather than generic pretrained attention.
AuthorsOpenReview ↗

Response to Reviewer CAFC (Part 2/2: Post-Processing Robustness on All & Unseen Generators)

W2 & Q2: Post-Processing Robustness on All and Unseen Generators

Following standard robustness evaluation practice (Wang et al., 2020), we train all compared detectors under Protocol I (SDv1.4 only) with Gaussian blur and JPEG augmentation, and evaluate 23 post-processing perturbations across all 13 test sets (1k1\text{k} real + 1k1\text{k} fake per set), including the 6 GAN families completely unseen during training (Sec. 4.5, Tab. 11; App. F, Tabs. 25–26).

Per-level JPEG & resizing AP (%) across all 13 Protocol I generators (App. F, Tab. 25):

Perturbation LevelOursEffortVIB-NetUnivFDC2P-CLIP
JPEG q=90q{=}9093.2390.1586.7983.8969.94
JPEG q=80q{=}8090.8986.5583.0680.5768.13
JPEG q=70q{=}7089.7084.2481.4878.9562.76
Resize 25%78.7373.2969.4665.7962.03
Resize 50%89.8182.8978.4274.0769.30
Resize 75%93.7187.3482.8877.7376.23

Mean AP (%) on unseen generator families (6 GAN test sets absent from training; App. F, Tab. 26):

Perturbation Category (Unseen GANs)OursEffortVIB-NetUnivFDC2P-CLIP
JPEG (q=90/80/70q{=}90/80/70)85.4081.2578.1869.2755.58
Resize (25/50/75%)86.0480.0775.0865.2766.66
Blur (σ=1/2/3\sigma{=}1/2/3)78.0172.1866.8260.6258.92
Social media pipeline73.3164.5568.3464.0853.67
Crop (3 settings)86.7783.4275.8471.5571.92
Screenshot (×1.25/×0.8\times 1.25/\times 0.8)90.8087.0080.9868.1375.73

Our detector achieves the highest AP at every JPEG quality and every resize ratio across all 13 sets, as well as the highest overall mean AP across all 23 perturbations (88.34% vs. Effort's 84.27%, Tab. 11). Crucially, on the 6 unseen GAN generators, our lead over the strongest CLIP-based baseline holds firmly across all perturbation types: +4.2 AP under JPEG, +6.0 AP under resizing, +5.8 AP under blur, and +5.0 AP under social-media processing.

I will keep my original score and look forward to seeing the suggested revisions reflected in the final version of the paper.

Reviewer_bPUvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies generalizable generated-image detection by reusing language-pretrained representations without textual input. It inserts a single pretrained LLM decoder layer between a frozen CLIP encoder and a binary classifier to re-encode all visual tokens under bidirectional attention. Experiments under three protocols covering GAN, diffusion, and autoregressive generators report strong cross-generator detection performance. Ablations suggest that both the pretrained LLM weights and interactions between the class and patch tokens contribute to the improvements.

Critical strengths
  1. The method is simple and computationally lighter than full MLLM-based forensic systems: it requires no text prompts or decoding and trains only about 4.3M parameters. The strong results across GAN, diffusion, and autoregressive generators demonstrate promising cross-generator generalization.

  2. The paper includes useful controlled ablations comparing linear probing, a capacity-matched MLP, class-token-only processing, a randomly initialized Transformer layer, different adaptation strengths, layer depths, and LLM backbones. These experiments provide evidence that the improvement cannot be explained solely by additional trainable capacity.

Critical weaknesses
  1. The results show improved classification and more foreground-focused activations, but do not establish that the model detects semantic or structural inconsistencies rather than residual local artifacts or generator-specific cues. Therefore, the paper’s central explanation for why the LLM layer improves generalization remains insufficiently validated.

  2. The evaluation omits common transformations such as JPEG compression, resizing, cropping, blur, and social-media recompression. Since generated images are frequently post-processed in practice, the method’s robustness and practical reliability remain unclear.

Questions for authors
  1. Can the authors evaluate the model on controlled examples containing explicit semantic or structural inconsistencies, or otherwise separate its reliance on high-level relations from low-level generator artifacts?

  2. How does performance change under JPEG compression, resizing, cropping, blur, and social-media-style recompression? Do the gains over CLIP-based baselines remain after these transformations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bPUv (Part 1/2: Separating High-Level Cues from Low-Level Artifacts)

We thank Reviewer bPUv for recognizing our method's simplicity, efficiency, cross-generator generalization, and controlled ablations. We reply in two parts (Part 1: W1/Q1; Part 2: W2/Q2); all new experiments appear in Sec. 4.5 (Tab. 11) and Appendices D.3 and F (Tabs. 19–21, 25–27) of the revised PDF (in blue).

W1 & Q1: Separating High-Level Relational Cues from Low-Level Artifacts

We do not claim our detector ignores low-level traces. Rather, it uses both low-level traces and higher-level content cues, while the frozen LLM layer reduces reliance on fragile generator-specific artifacts. To separate them, we run three controlled interventions on Protocol I (App. D.3), comparing our model against three internal controls sharing the exact same frozen CLIP ViT-L/14 tokens (trained without data augmentation so robustness training does not confound mechanism analysis): linear probing, tcls\boldsymbol{t}_{\text{cls}}-only LLM re-encoding, and uniform-attention LLM re-encoding (weights fixed to 1/(N+1)1/(N{+}1)).

1. Suppressing low-level artifacts via low-pass filtering (App. D.3, Tab. 19). Upsampling and noise residuals concentrate in high frequencies (Frank et al., 2020; Liu et al., 2022). If the LLM layer's gain came solely from low-level artifacts, its lead over linear probing should vanish under Gaussian blur or down-up resizing.

Variant (Δ\DeltaAP vs. clean)Blur σ=1\sigma{=}1Blur σ=2\sigma{=}2Blur σ=3\sigma{=}3Resize ×0.5\times 0.5Resize ×0.25\times 0.25
CLIP + linear probe−13.03-13.03−26.94-26.94−34.09-34.09−14.98-14.98−26.01-26.01
LLM layer, tcls\boldsymbol{t}_{\text{cls}} only−16.43-16.43−30.06-30.06−36.21-36.21−17.88-17.88−30.12-30.12
LLM layer, uniform attention−16.91-16.91−30.27-30.27−37.01-37.01−18.51-18.51−31.18-31.18
Ours (full model)−10.57-10.57−21.18-21.18−28.36-28.36−11.72-11.72−22.52-22.52

All variants drop, confirming high-frequency cues help every detector. Yet our full model degrades least in every setting, widening its margin over the three controls by 2.5–9.1 AP points after filtering, whereas the two restricted LLM variants drop 2.1–5.2 points more than linear probing. Thus, the gain tied to content-weighted attention is precisely what survives low-pass filtering.

2. Disrupting spatial layout while preserving local patch artifacts (App. D.3, Tab. 20). Conversely, we split the 224×224224{\times}224 input into a g×gg{\times}g grid and randomly permute tiles (at 16×1616{\times}16, every ViT patch is displaced).

Variant (Δ\DeltaAP vs. clean)2×22{\times}24×44{\times}48×88{\times}816×1616{\times}16
CLIP + linear probe−1.78-1.78−4.86-4.86−7.62-7.62−12.23-12.23
LLM layer, tcls\boldsymbol{t}_{\text{cls}} only−0.67-0.67−1.87-1.87−3.92-3.92−8.63-8.63
LLM layer, uniform attention−1.60-1.60−2.88-2.88−5.14-5.14−10.55-10.55
Ours (full model)−0.40-0.40−0.83-0.83−1.80-1.80−5.07-5.07

Disrupting global layout hurts far less than removing high frequencies, and our model again drops least (−5.07-5.07 AP at 16×1616{\times}16). Since RoPE is disabled, tcls\boldsymbol{t}_{\text{cls}} queries patches by content rather than 2D coordinates, still aggregating local evidence after shuffling.

3. Varying semantic plausibility under matched generation traces (App. D.3, Tab. 21). For 600 real images, we inpaint the central quarter twice using the same model, mask, and initial noise seed: once with an original-class object (coherent) and once with the most semantically distant class (incoherent). We report the fraction of pairs where the incoherent edit receives a higher fake score (0.50.5 = no preference), before and after σ=1\sigma{=}1 blur.

Variant (Pairwise preference)UnblurredBlur σ=1\sigma{=}1
CLIP + linear probe0.400.53
LLM layer, tcls\boldsymbol{t}_{\text{cls}} only0.670.60
Ours (full model)0.720.63

Linear probing leans opposite (0.400.40) and is near chance after blur (0.530.53), confirming low-level inpainting traces do not favor incoherent edits. Adding the LLM layer ranks the incoherent edit as more fake in 67–72% of pairs (60–63% after blur), directly isolating sensitivity to semantic plausibility.

AuthorsOpenReview ↗

Response to Reviewer bPUv (Part 2/2: Robustness to Image Post-Processing)

W2 & Q2: Robustness to Image Post-Processing

In Sec. 4.5 (Tab. 11) and App. F (Tabs. 25–27), we evaluate 23 perturbations across all 13 Protocol I test sets (1k1\text{k} real + 1k1\text{k} fake per set): JPEG (q∈q \in {90, 80, 70}), resize (25/50/75%), blur (σ∈\sigma \in {1, 2, 3}), JPEG$\to$blur (q90→σ1q90\to\sigma1, q70→σ2q70\to\sigma2), Gaussian noise (std ∈\in {5, 10, 20}), color jitter (brightness/contrast 0.4, hue 0.1), social media (long-side 1080→×0.75→1080\to\times 0.75\toJPEG q75q75), crop (random 75%/50%, center 50%), and screenshot scaling (×1.25/×0.8\times 1.25/\times 0.8). Following standard robustness practice (Wang et al., 2020), all detectors are trained with blur and JPEG augmentation.

Perturbation (Mean AP %, 13 sets)OursEffortVIB-NetUnivFDC2P-CLIP
Clean (aug-trained, 1k+1k split)97.2896.5295.1185.9790.68
JPEG (q=90/80/70q{=}90/80/70)91.2886.9883.7881.1466.94
Resize (25/50/75%)87.4281.1876.9272.5369.19
Blur (σ=1/2/3\sigma{=}1/2/3)81.8276.0672.4967.1063.95
JPEG →\to Blur (2 settings)83.1176.9573.7569.0563.98
Noise (std=5/10/20\text{std}{=}5/10/20)87.3881.3580.2874.2370.65
Color jitter (3 settings)96.2794.3793.1983.5188.18
Social media79.0074.5977.5979.9067.12
Crop (3 settings)90.2590.0584.0883.9486.26
Screenshot (×1.25/×0.8\times 1.25/\times 0.8)91.6689.8884.7981.9885.81
Mean over 23 perturbations (AP)88.3484.2781.1776.9374.01
Mean over 23 perturbations (ACC)75.7473.3275.0061.9664.05

Averaged over all 23 perturbations, our detector ranks 1st in both mean AP (88.34% vs. Effort's 84.27%) and mean ACC (75.74% vs. VIB-Net's 75.00%), achieving the top AP on 20/23 settings (+3.1 to +7.1 AP across all JPEG, resize, blur, and JPEG$\to$blur levels). Under noise/color jitter, our AP stays highest while fixed-threshold scores shift slightly. Under crop/screenshot, Effort and our model perform comparably.

I thank the authors for the detailed response and the additional experiments.

The controlled interventions on filtering, layout permutation and semantic plausibility separate the model's reliance on high-level content cues from low-level generator artifacts, which supports the paper's central claim. The evaluation over a wide range of post-processing perturbations across all Protocol I test sets also resolves my robustness concern.

I maintain my original rating of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Manuscript Revisions

We sincerely thank the Area Chair and all three reviewers (bPUv, CAFC, PpTn) for their constructive feedback and for recognizing our work's promising cross-modal transfer direction, simplicity and parameter efficiency (~4.3M trainable parameters, no text input), cross-generator generalization, and controlled ablations.

We have uploaded a revised PDF (revisions highlighted in blue) adding post-processing robustness to the main text (Sec. 4.5) and detailed mechanistic, baseline, and calibration analyses to Appendices D–F:

  1. Post-Processing Robustness in Main Text (bPUv W2/Q2, CAFC W2/Q2, PpTn Q3 — Sec. 4.5, Tab. 11 & App. F, Tabs. 25–28): Across 23 perturbations on all 13 Protocol I test sets, our detector ranks 1st in both mean AP (88.34%) and mean ACC (75.74%), winning 20/23 settings in AP and maintaining clear margins on unseen GANs.
  2. Language-Specific Priors vs. Visual/Multimodal Blocks (CAFC W1/Q1, PpTn W1/W2 — App. D.1, Tab. 16): Replacing our LLM block with capacity-matched pretrained visual (DINOv2-g) and multimodal (EVA-CLIP-8B, MetaCLIP-H) blocks on identical CLIP tokens shows language blocks (TinyLlama, Llama-3.1-8B, BERT-large) consistently outperform both by +0.8 to +4.0 ACC and +0.9 to +3.0 AP.
  3. Pang et al. Baselines & Relational Mechanism (PpTn W1/Q2, W2, bPUv W1/Q1 — App. D.2–D.3, Tabs. 17–21): Reproducing Pang et al. (2024) drops accuracy by 7.71–28.23 ACC due to shortcut overfitting on unseen GANs. Uniform-attention controls, low-pass filtering, patch shuffling, and noise-matched semantic inpainting isolate content-based attention and semantic plausibility from low-level artifacts.
  4. Recent Detectors, Low-FPR & Calibration (PpTn W3/Q1, Q4 — App. E, Tabs. 22–24): We evaluate C2P-CLIP, B-Free, AntifakePrompt, and Community Forensics, and show +10.30% TPR@1%FPR (+17.89% on unseen GANs) over Effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.