Aligning Beyond Semantics: Learning Desemanticized Anchors for AI-Generated Image Detection

CLIP-based detectors have advanced AI-generated image detection by transferring rich visual representations learned through large-scale image-text pretraining, yet their semantic strengths do not always translate into reliable forgery discrimination. We observe an underappreciated asymmetry on out-of-domain data: the highly competitive performance of a CLIP-adapted detector is driven largely by an exceptionally strong preference for real-image predictions, while its fake-image identification is substantially worse than that of traditional artifact-oriented detectors. To investigate this imbalance, we design a semantic-swap diagnostic that reverses the training-time correlation between image-content semantics and real/fake labels. The evaluated detectors’ predictions often follow image content even when it conflicts with an image’s real/fake status. We term this Semantic Eclipsing: a failure mode in which semantic cues remain in CLIP’s latent space, overshadowing artifact-relevant evidence. In this work, we propose Semantically Unconstrained Alignment (SUA) to suppress semantic eclipsing. Rather than defining alignment targets through fixed semantic descriptions, SUA learns class-specific desemanticized anchors on CLIP’s text side through image-anchor contrastive learning. Extensive experiments across multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches, particularly under challenging cross-semantic variations.