- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies AI generated image detection. The authors point out that existing CLIP based detectors achieve high overall metrics largely because they predict many images as real. Their fake image recognition is actually weaker than traditional artifact oriented detectors. They call this phenomenon Semantic Eclipsing. To verify this hypothesis they design a semantic swap stress test. During training real and fake labels are artificially bound to semantic categories. At test time the semantic pairs are swapped for example Fake Horse versus Real Car. Baseline CLIP detectors show very high semantic discriminability up to 94.25% but very low artifact discriminability as low as 5.75%. This indicates they rely mainly on semantic shortcuts rather than forgery evidence. To address this the authors propose Semantically Unconstrained Alignment or SUA. Specifically they learn continuous learnable class specific anchors on the CLIP text side called desemanticized textual anchors. Each class anchor is formed by concatenating a fixed SOS embedding a learnable token and a fixed EOT embedding. Then a contrastive loss and a classification loss are combined as the total loss.
Critical strengths
1: The paper points out that existing CLIP based AI generated image detectors despite good overall accuracy owe much of this advantage to a strong preference for predicting real images. This observation is important because many prior works only look at overall accuracy which can easily hide poor fake image recall.
2: SUA does not introduce a very large new network. Instead it adds a small number of learnable tokens on the CLIP text side and uses image anchor contrastive learning to adjust the visual representation. At inference the text encoder and contrastive branch can be discarded leaving only the visual encoder and a classification head. This makes deployment relatively lightweight.
3: The paper evaluates on multiple datasets including UniverForen GenImage CSAIID Community Forensics and SPAI. It covers cross generator cross semantic cross dataset and cross resolution settings. This to some extent supports the effectiveness of the method.
Critical weaknesses
1: The method lacks novelty. SUA uses continuous learnable tokens concatenates SOS and EOT and passes them through a frozen text encoder to obtain anchors. This is very close to CoOP and prompt tuning with continuous context tokens. The paper does not add explicit desemantic regularization orthogonal constraints or adversarial semantic disentanglement. The training objective is only image anchor contrastive loss plus BCE.
2: The Semantic Eclipsing diagnosis is somewhat artificial and does not directly prove it is the main cause of real out of distribution failures. In the semantic swap experiment training semantics and real and fake labels are fully coupled and then completely reversed at test time. This shows CLIP detectors can exploit semantic shortcuts but in real data semantics and real and fake labels are not strictly bound. The paper does not quantify how much semantic shortcuts contribute on real benchmarks. It also does not convincingly show that Semantic Eclipsing is the main reason for poor cross semantic performance.
3: The method is not very interpretable. Figure 4 shows anchor tokens near meaningless symbols and subwords but the paper cannot explain what forgery features each token encodes.
4: There are many hyperparameters and the discussion of hyperparameter sensitivity is insufficient.
Questions for authors
1: The paper explicitly states that at inference the text encoder and contrastive branch are discarded and only the classification head is used. In Table 5 AnchorInf achieves only 82.08% while the full method reaches 93.15%. This is even lower than CLIP LoRA at 88.00%. Does this mean the anchors themselves have almost no discriminative power at inference. If anchors do not participate at inference at all what is the essential difference between SUA and LoRA plus BCE classification head.
2: In the paper SUA anchors are formed by concatenating continuous learnable tokens with SOS and EOT but the training objective is only image anchor contrastive loss and BCE. Figure 4 shows anchor nearest neighbors are meaningless symbols and subwords. However continuous tokens passed through a frozen text encoder can naturally fall in sparse regions of the discrete vocabulary. This does not prove that anchors are disentangled from content semantics. What is the essential difference between SUA and CoOP or prompt tuning.
3: Figure 4 uses t-SNE to show learned real and fake anchors and their nearest words. It claims this proves forgery traces are decoupled from semantic concepts. But at inference the anchors are discarded and the final prediction comes from the classification head. How does the anchor visualization explain the final classification.