acceptodds
Withdrawn from ICLR 2027

Aligning Beyond Semantics: Learning Desemanticized Anchors for AI-Generated Image Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

CLIP-based detectors have advanced AI-generated image detection by transferring rich visual representations learned through large-scale image-text pretraining, yet their semantic strengths do not always translate into reliable forgery discrimination. We observe an underappreciated asymmetry on out-of-domain data: the highly competitive performance of a CLIP-adapted detector is driven largely by an exceptionally strong preference for real-image predictions, while its fake-image identification is substantially worse than that of traditional artifact-oriented detectors. To investigate this imbalance, we design a semantic-swap diagnostic that reverses the training-time correlation between image-content semantics and real/fake labels. The evaluated detectors’ predictions often follow image content even when it conflicts with an image’s real/fake status. We term this Semantic Eclipsing: a failure mode in which semantic cues remain in CLIP’s latent space, overshadowing artifact-relevant evidence. In this work, we propose Semantically Unconstrained Alignment (SUA) to suppress semantic eclipsing. Rather than defining alignment targets through fixed semantic descriptions, SUA learns class-specific desemanticized anchors on CLIP’s text side through image-anchor contrastive learning. Extensive experiments across multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches, particularly under challenging cross-semantic variations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Media Forensics,Visual Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Fy5WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies AI generated image detection. The authors point out that existing CLIP based detectors achieve high overall metrics largely because they predict many images as real. Their fake image recognition is actually weaker than traditional artifact oriented detectors. They call this phenomenon Semantic Eclipsing. To verify this hypothesis they design a semantic swap stress test. During training real and fake labels are artificially bound to semantic categories. At test time the semantic pairs are swapped for example Fake Horse versus Real Car. Baseline CLIP detectors show very high semantic discriminability up to 94.25% but very low artifact discriminability as low as 5.75%. This indicates they rely mainly on semantic shortcuts rather than forgery evidence. To address this the authors propose Semantically Unconstrained Alignment or SUA. Specifically they learn continuous learnable class specific anchors on the CLIP text side called desemanticized textual anchors. Each class anchor is formed by concatenating a fixed SOS embedding a learnable token and a fixed EOT embedding. Then a contrastive loss and a classification loss are combined as the total loss.

Critical strengths

1: The paper points out that existing CLIP based AI generated image detectors despite good overall accuracy owe much of this advantage to a strong preference for predicting real images. This observation is important because many prior works only look at overall accuracy which can easily hide poor fake image recall.

2: SUA does not introduce a very large new network. Instead it adds a small number of learnable tokens on the CLIP text side and uses image anchor contrastive learning to adjust the visual representation. At inference the text encoder and contrastive branch can be discarded leaving only the visual encoder and a classification head. This makes deployment relatively lightweight.

3: The paper evaluates on multiple datasets including UniverForen GenImage CSAIID Community Forensics and SPAI. It covers cross generator cross semantic cross dataset and cross resolution settings. This to some extent supports the effectiveness of the method.

Critical weaknesses

1: The method lacks novelty. SUA uses continuous learnable tokens concatenates SOS and EOT and passes them through a frozen text encoder to obtain anchors. This is very close to CoOP and prompt tuning with continuous context tokens. The paper does not add explicit desemantic regularization orthogonal constraints or adversarial semantic disentanglement. The training objective is only image anchor contrastive loss plus BCE.

2: The Semantic Eclipsing diagnosis is somewhat artificial and does not directly prove it is the main cause of real out of distribution failures. In the semantic swap experiment training semantics and real and fake labels are fully coupled and then completely reversed at test time. This shows CLIP detectors can exploit semantic shortcuts but in real data semantics and real and fake labels are not strictly bound. The paper does not quantify how much semantic shortcuts contribute on real benchmarks. It also does not convincingly show that Semantic Eclipsing is the main reason for poor cross semantic performance.

3: The method is not very interpretable. Figure 4 shows anchor tokens near meaningless symbols and subwords but the paper cannot explain what forgery features each token encodes.

4: There are many hyperparameters and the discussion of hyperparameter sensitivity is insufficient.

Questions for authors

1: The paper explicitly states that at inference the text encoder and contrastive branch are discarded and only the classification head is used. In Table 5 AnchorInf achieves only 82.08% while the full method reaches 93.15%. This is even lower than CLIP LoRA at 88.00%. Does this mean the anchors themselves have almost no discriminative power at inference. If anchors do not participate at inference at all what is the essential difference between SUA and LoRA plus BCE classification head.

2: In the paper SUA anchors are formed by concatenating continuous learnable tokens with SOS and EOT but the training objective is only image anchor contrastive loss and BCE. Figure 4 shows anchor nearest neighbors are meaningless symbols and subwords. However continuous tokens passed through a frozen text encoder can naturally fall in sparse regions of the discrete vocabulary. This does not prove that anchors are disentangled from content semantics. What is the essential difference between SUA and CoOP or prompt tuning.

3: Figure 4 uses t-SNE to show learned real and fake anchors and their nearest words. It claims this proves forgery traces are decoupled from semantic concepts. But at inference the anchors are discarded and the final prediction comes from the classification head. How does the anchor visualization explain the final classification.

Reviewer_n4QvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cross-domain generalization of CLIP-based detectors for AI-generated images. The authors observe that the strong overall performance of existing CLIP-based detectors is often driven by near-perfect recognition of real images, while unseen generated images are frequently misclassified as real. Through a semantic-swap stress test, the paper identifies Semantic Eclipsing, a failure mode in which high-level content semantics dominate lower-level evidence of generation artifacts. To mitigate this issue, the authors propose Semantically Unconstrained Alignment (SUA), which constructs real and fake textual anchors using two sets of learnable continuous tokens and employs image–anchor contrastive learning to adapt the CLIP image encoder through LoRA. Experiments on cross-generator, cross-semantic, cross-dataset, and cross-resolution generalization show competitive performance across several benchmarks. The textual branch can be removed after training, so SUA introduces no additional inference-time overhead.

Critical strengths
  • The problem motivation is clear and supported by a useful diagnostic experiment. The semantic-swap stress test trains detectors on Real-Horse versus Fake-Car images and evaluates them on Fake-Horse versus Real-Car images. This controlled reversal helps distinguish whether a detector relies primarily on object semantics or on evidence related to image generation. The analysis also reveals that the apparently strong performance of some CLIP-based detectors is largely attributable to their near-perfect real-image accuracy.
  • The proposed method is simple and practical. Rather than using semantically explicit descriptions such as “a real image” and “a fake image,” SUA learns two sets of continuous tokens and maps them into class-specific anchors through the frozen CLIP text encoder. The design is easy to integrate with a LoRA-adapted CLIP image encoder, and the text branch is required only during training.
  • The empirical evaluation covers several important forms of distribution shift. The paper evaluates cross-generator, cross-semantic, cross-dataset, and cross-resolution generalization on multiple datasets and against a broad set of baselines. These experiments provide a relatively comprehensive assessment of detector generalization.
Critical weaknesses
  • The paper does not report variability across independent training runs or statistical significance. Some improvements are very small. For example, on the cross-resolution benchmark, SUA achieves 82.6% balanced accuracy and 90.2% AP, compared with 82.3% and 90.1% for SPAI. Without results over multiple random seeds, standard deviations, or confidence intervals, it is difficult to determine whether such differences are stable and meaningful.
  • Generalization to several recent generators remains weak. In the cross-resolution experiment, SUA obtains balanced accuracies of only 58.5%, 48.8%, and 48.9% on FLUX, SD3, and DALL-E 3, respectively. Thus, although the average performance is strong, the detector does not appear to learn forgery cues that transfer reliably across all generation mechanisms. The aggregate metric obscures these substantial generator-specific failures.
  • The causal connection between Semantic Eclipsing and the proposed solution is not sufficiently established. Operationally, SUA resembles class-specific continuous prompt learning combined with contrastive regularization. The ablations against CoOP, CLIP-LoRA, and WoTextEnc indicate that the individual design choices are useful, but they do not demonstrate that the performance gain specifically results from reduced reliance on semantic content. Likewise, the observation that learned tokens are close to subword fragments or meaningless symbols in a two-dimensional t-SNE visualization is not sufficient evidence that the anchors encode generalizable forgery traces. More direct measurements of semantic information and artifact sensitivity in the learned representation would strengthen the central explanation.
Questions for authors
  • Are the results stable across different initializations and training runs? The optimization of continuous tokens may be sensitive to initialization and local optima. Could the authors report mean ± standard deviation over multiple runs and quantitatively compare anchors learned from different training datasets and random seeds, rather than relying primarily on t-SNE visualizations?
  • How does the method perform on partially AI-edited images? The current experiments mainly consider fully generated images. It would be valuable to evaluate localized AI manipulations, such as inpainting, object replacement, or other region-level edits, to determine whether the learned anchors generalize when generation artifacts occupy only a small part of an otherwise real image.
Reviewer_71q8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper highlights a limitation of existing CLIP-based detectors, which achieve competitive overall detection performance on out-of-domain data largely due to a strong preference for real-image predictions, while performing substantially worse than traditional artifact-oriented detectors in fake-image identification. The paper characterizes this imbalance as Semantic Eclipsing, a phenomenon in which semantic cues overshadow artifact-relevant evidence. To mitigate this issue, the paper proposes Semantically Unconstrained Alignment (SUA), which learns class-specific desemanticized anchors to reduce semantic dependence. The proposed method achieves strong performance across multiple generalization benchmarks, and its key design choices are further validated through ablation studies.

Critical strengths
  • SUA achieves strong image detection performance compared to existing methods. In particular, it demonstrates consistent performance improvements under various distribution shifts involving generative models, image semantics, datasets, and resolutions, providing broad empirical evidence of its generalization capability.

  • The paper conducts a systematic ablation study comparing key components and training strategies to analyze their impact on detection performance. These experiments provide empirical support for the key design choices of the proposed method.

Critical weaknesses
  • The paper presents Semantic Eclipsing as a major failure mode of CLIP-based detectors, but the Semantic-Swap Stress Test evaluates only a limited number of detectors, making it difficult to establish the generality of this phenomenon. Furthermore, the claim that Semantic Eclipsing originates from the inherent properties of pretrained VLMs is not sufficiently supported.

  • Although SUA demonstrates strong performance across various generalization settings, its evaluation relies on a single CLIP ViT-L/14 backbone, leaving its generalizability to other backbones unclear. Moreover, the Anchor Visualization alone provides limited evidence that SUA actually reduces reliance on semantic cues and increases the use of artifact-relevant evidence during detection.

Questions for authors

The following questions are related to the critical weaknesses identified above.

  • Does Semantic Eclipsing consistently occur in other CLIP-based detectors beyond DeFake and C2P-CLIP? Additionally, is there further evidence to distinguish whether this phenomenon originates from the inherent properties of pretrained VLMs or from fine-tuning and semantic biases in the training data?

  • Does SUA maintain its performance improvements with backbones other than CLIP ViT-L/14? Could the effectiveness of the proposed method be demonstrated across different backbone sizes or pretraining strategies?

  • The paper claims that SUA reduces reliance on semantic cues and increases the use of artifact-relevant evidence. However, Anchor Visualization alone provides limited direct evidence for this claim. Could the paper provide quantitative analyses or additional experimental results demonstrating that these changes occur during detection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.